Oct 30 20:57:26.695307 ip-10-0-142-3 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 20:57:26.695318 ip-10-0-142-3 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 20:57:26.695326 ip-10-0-142-3 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 20:57:26.695581 ip-10-0-142-3 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 20:57:38.083001 ip-10-0-142-3 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 20:57:38.083019 ip-10-0-142-3 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 28d25c4675df49969cc7b0820c83cb76 -- Oct 30 20:59:31.774301 ip-10-0-142-3 systemd[1]: Starting Kubernetes Kubelet... Oct 30 20:59:32.140086 ip-10-0-142-3 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:59:32.140086 ip-10-0-142-3 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 20:59:32.140086 ip-10-0-142-3 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:59:32.140086 ip-10-0-142-3 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 20:59:32.147587 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147508 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:59:32.147587 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147586 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:59:32.147587 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147591 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:59:32.147676 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147594 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:59:32.147676 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147597 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:59:32.147676 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147600 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:59:32.147676 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147603 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:59:32.147676 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147606 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:59:32.147676 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147608 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:59:32.147676 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147616 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:59:32.147676 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147619 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:59:32.147676 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147622 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:59:32.147676 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147624 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:59:32.147676 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147627 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:59:32.147676 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147629 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:59:32.147676 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147632 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:59:32.147676 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147635 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:59:32.147676 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147637 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:59:32.147676 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147640 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:59:32.147676 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147642 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:59:32.147676 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147645 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:59:32.147676 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147647 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:59:32.147676 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147650 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:59:32.148149 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147652 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:59:32.148149 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147655 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:59:32.148149 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147657 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:59:32.148149 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147660 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:59:32.148149 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147662 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:59:32.148149 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147665 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:59:32.148149 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147668 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:59:32.148149 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147670 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:59:32.148149 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147673 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:59:32.148149 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147676 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:59:32.148149 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147678 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:59:32.148149 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147681 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:59:32.148149 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147684 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:59:32.148149 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147686 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:59:32.148149 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147689 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:59:32.148149 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147692 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:59:32.148149 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147695 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:59:32.148149 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147697 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:59:32.148149 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147700 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:59:32.148149 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147702 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:59:32.148613 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147704 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:59:32.148613 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147707 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:59:32.148613 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147709 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:59:32.148613 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147712 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:59:32.148613 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147714 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:59:32.148613 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147717 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:59:32.148613 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147719 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:59:32.148613 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147722 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:59:32.148613 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147724 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:59:32.148613 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147727 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:59:32.148613 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147729 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:59:32.148613 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147731 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:59:32.148613 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147734 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:59:32.148613 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147737 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:59:32.148613 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147739 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:59:32.148613 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147742 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:59:32.148613 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147746 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:59:32.148613 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147749 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:59:32.148613 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147752 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:59:32.149048 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147754 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:59:32.149048 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147757 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:59:32.149048 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147760 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:59:32.149048 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147762 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:59:32.149048 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147764 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:59:32.149048 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147767 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:59:32.149048 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147770 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:59:32.149048 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147772 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:59:32.149048 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147775 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:59:32.149048 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147777 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:59:32.149048 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147780 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:59:32.149048 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147782 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:59:32.149048 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147785 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:59:32.149048 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147787 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:59:32.149048 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147790 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:59:32.149048 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147792 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:59:32.149048 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147795 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:59:32.149048 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147797 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:59:32.149048 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147800 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:59:32.149486 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147803 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:59:32.149486 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147805 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:59:32.149486 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147807 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:59:32.149486 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147810 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:59:32.149486 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147812 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:59:32.149486 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147815 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:59:32.149486 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147817 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:59:32.149486 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147820 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:59:32.149486 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147824 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:59:32.149486 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147828 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:59:32.149486 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147831 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:59:32.149486 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147834 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:59:32.149486 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147837 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:59:32.149486 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147840 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:59:32.149486 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147842 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:59:32.149486 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147847 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:59:32.149486 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147850 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:59:32.149486 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147853 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:59:32.149486 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147855 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:59:32.149926 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147857 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:59:32.149926 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147860 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:59:32.149926 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147862 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:59:32.149926 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147865 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:59:32.149926 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147867 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:59:32.149926 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.147870 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:59:32.149926 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148628 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:59:32.149926 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148635 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:59:32.149926 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148637 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:59:32.149926 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148640 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:59:32.149926 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148643 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:59:32.149926 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148646 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:59:32.149926 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148650 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:59:32.149926 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148654 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:59:32.149926 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148656 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:59:32.149926 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148659 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:59:32.149926 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148662 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:59:32.149926 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148665 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:59:32.149926 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148668 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:59:32.150397 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148670 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:59:32.150397 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148673 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:59:32.150397 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148675 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:59:32.150397 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148678 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:59:32.150397 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148680 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:59:32.150397 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148683 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:59:32.150397 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148685 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:59:32.150397 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148688 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:59:32.150397 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148690 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:59:32.150397 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148693 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:59:32.150397 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148695 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:59:32.150397 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148698 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:59:32.150397 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148701 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:59:32.150397 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148703 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:59:32.150397 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148705 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:59:32.150397 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148708 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:59:32.150397 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148710 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:59:32.150397 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148713 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:59:32.150397 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148715 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:59:32.150834 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148718 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:59:32.150834 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148720 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:59:32.150834 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148723 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:59:32.150834 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148725 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:59:32.150834 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148728 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:59:32.150834 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148731 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:59:32.150834 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148734 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:59:32.150834 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148736 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:59:32.150834 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148739 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:59:32.150834 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148741 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:59:32.150834 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148744 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:59:32.150834 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148746 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:59:32.150834 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148749 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:59:32.150834 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148751 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:59:32.150834 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148754 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:59:32.150834 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148756 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:59:32.150834 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148759 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:59:32.150834 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148761 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:59:32.150834 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148764 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:59:32.150834 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148766 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:59:32.151304 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148768 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:59:32.151304 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148771 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:59:32.151304 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148773 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:59:32.151304 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148775 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:59:32.151304 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148778 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:59:32.151304 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148781 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:59:32.151304 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148783 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:59:32.151304 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148785 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:59:32.151304 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148788 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:59:32.151304 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148790 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:59:32.151304 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148793 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:59:32.151304 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148795 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:59:32.151304 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148797 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:59:32.151304 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148800 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:59:32.151304 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148803 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:59:32.151304 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148805 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:59:32.151304 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148807 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:59:32.151304 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148810 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:59:32.151304 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148813 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:59:32.151304 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148815 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:59:32.151788 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148818 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:59:32.151788 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148821 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:59:32.151788 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148824 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:59:32.151788 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148826 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:59:32.151788 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148831 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:59:32.151788 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148833 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:59:32.151788 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148836 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:59:32.151788 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148838 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:59:32.151788 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148841 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:59:32.151788 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148843 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:59:32.151788 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148846 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:59:32.151788 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148849 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:59:32.151788 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148852 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:59:32.151788 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148854 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:59:32.151788 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148857 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:59:32.151788 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148859 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:59:32.151788 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148862 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:59:32.151788 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148864 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:59:32.151788 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148867 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:59:32.152237 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148869 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:59:32.152237 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148872 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:59:32.152237 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148874 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:59:32.152237 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148877 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:59:32.152237 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148879 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:59:32.152237 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148881 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:59:32.152237 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148884 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:59:32.152237 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148886 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:59:32.152237 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148889 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:59:32.152237 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148893 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:59:32.152237 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148897 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:59:32.152237 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148900 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:59:32.152237 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148903 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:59:32.152237 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148906 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:59:32.152237 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.148908 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:59:32.152237 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149375 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:59:32.152237 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149380 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:59:32.152237 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149385 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:59:32.152237 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149387 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:59:32.152882 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149390 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:59:32.152882 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149392 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:59:32.152882 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149395 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:59:32.152882 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149397 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:59:32.152882 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149400 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:59:32.152882 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149403 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:59:32.152882 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149405 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:59:32.152882 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149408 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:59:32.152882 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149411 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:59:32.152882 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149413 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:59:32.152882 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149415 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:59:32.152882 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149418 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:59:32.152882 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149420 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:59:32.152882 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149423 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:59:32.152882 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149425 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:59:32.152882 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149428 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:59:32.152882 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149430 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:59:32.152882 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149433 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:59:32.152882 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149435 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:59:32.153539 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149438 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:59:32.153539 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149441 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:59:32.153539 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149443 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:59:32.153539 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149446 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:59:32.153539 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149449 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:59:32.153539 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149452 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:59:32.153539 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149454 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:59:32.153539 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149457 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:59:32.153539 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149459 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:59:32.153539 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149462 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:59:32.153539 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149464 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:59:32.153539 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149468 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:59:32.153539 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149471 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:59:32.153539 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149475 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:59:32.153539 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149478 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:59:32.153539 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149482 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:59:32.153539 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149485 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:59:32.153539 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149488 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:59:32.153539 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149490 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:59:32.153991 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149493 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:59:32.153991 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149496 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:59:32.153991 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149500 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:59:32.153991 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149502 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:59:32.153991 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149505 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:59:32.153991 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149507 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:59:32.153991 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149510 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:59:32.153991 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149512 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:59:32.153991 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149515 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:59:32.153991 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149517 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:59:32.153991 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149519 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:59:32.153991 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149522 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:59:32.153991 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149524 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:59:32.153991 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149527 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:59:32.153991 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149529 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:59:32.153991 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149532 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:59:32.153991 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149535 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:59:32.153991 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149537 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:59:32.153991 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149540 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:59:32.153991 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149543 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:59:32.154574 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149545 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:59:32.154574 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149547 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:59:32.154574 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149550 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:59:32.154574 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149552 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:59:32.154574 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149556 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:59:32.154574 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149558 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:59:32.154574 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149561 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:59:32.154574 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149563 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:59:32.154574 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149566 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:59:32.154574 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149569 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:59:32.154574 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149571 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:59:32.154574 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149574 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:59:32.154574 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149577 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:59:32.154574 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149581 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:59:32.154574 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149585 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:59:32.154574 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149587 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:59:32.154574 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149590 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:59:32.154574 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149592 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:59:32.154574 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149595 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:59:32.154574 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149597 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:59:32.155082 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149599 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:59:32.155082 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149602 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:59:32.155082 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149604 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:59:32.155082 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149607 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:59:32.155082 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149609 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:59:32.155082 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149612 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:59:32.155082 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149614 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:59:32.155082 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149617 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:59:32.155082 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149620 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:59:32.155082 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149622 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:59:32.155082 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149625 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:59:32.155082 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149627 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:59:32.155082 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149629 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:59:32.155082 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149632 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:59:32.155082 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149634 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:59:32.155082 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149637 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:59:32.155082 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149641 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:59:32.155082 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149643 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:59:32.155082 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149645 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149648 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149650 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149653 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149655 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.149658 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151364 2577 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151375 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151383 2577 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151388 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151392 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151395 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151400 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151404 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151407 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151410 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151413 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151416 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151419 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151422 2577 flags.go:64] FLAG: --cgroup-root="" Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151425 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151428 2577 flags.go:64] FLAG: --client-ca-file="" Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151431 2577 flags.go:64] FLAG: --cloud-provider="external" Oct 30 20:59:32.155511 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151435 2577 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151440 2577 flags.go:64] FLAG: --cluster-domain="" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151443 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151446 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151449 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151452 2577 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151456 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151459 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151462 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151465 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151468 2577 flags.go:64] FLAG: --contention-profiling="false" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151471 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151474 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151477 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151480 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151485 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151488 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151491 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151494 2577 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151497 2577 flags.go:64] FLAG: --enable-server="true" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151500 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151504 2577 flags.go:64] FLAG: --event-burst="100" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151507 2577 flags.go:64] FLAG: --event-qps="50" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151510 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 20:59:32.156059 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151513 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151516 2577 flags.go:64] FLAG: --eviction-hard="" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151520 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151522 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151525 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151528 2577 flags.go:64] FLAG: --eviction-soft="" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151531 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151534 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151537 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151540 2577 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151543 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151546 2577 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151549 2577 flags.go:64] FLAG: --feature-gates="" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151553 2577 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151556 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151560 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151563 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151566 2577 flags.go:64] FLAG: --healthz-port="10248" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151569 2577 flags.go:64] FLAG: --help="false" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151572 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-142-3.ec2.internal" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151575 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151578 2577 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151581 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151584 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 20:59:32.156646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151588 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151591 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151593 2577 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151596 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151599 2577 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151602 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151605 2577 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151608 2577 flags.go:64] FLAG: --kube-reserved="" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151610 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151613 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151616 2577 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151619 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151622 2577 flags.go:64] FLAG: --lock-file="" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151625 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151628 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151631 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151637 2577 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151639 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151643 2577 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151646 2577 flags.go:64] FLAG: --logging-format="text" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151649 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151652 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151655 2577 flags.go:64] FLAG: --manifest-url="" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151658 2577 flags.go:64] FLAG: --manifest-url-header="" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151662 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 20:59:32.157198 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151665 2577 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151669 2577 flags.go:64] FLAG: --max-pods="110" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151672 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151675 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151678 2577 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151681 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151684 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151687 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151690 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151696 2577 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151699 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151702 2577 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151705 2577 flags.go:64] FLAG: --pod-cidr="" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151708 2577 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151710 2577 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151713 2577 flags.go:64] FLAG: --pods-per-core="0" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151716 2577 flags.go:64] FLAG: --port="10250" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151719 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151722 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0953d1f9b975c2ff4" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151725 2577 flags.go:64] FLAG: --qos-reserved="" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151728 2577 flags.go:64] FLAG: --read-only-port="10255" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151731 2577 flags.go:64] FLAG: --register-node="true" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151734 2577 flags.go:64] FLAG: --register-with-taints="" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151737 2577 flags.go:64] FLAG: --registry-burst="10" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151740 2577 flags.go:64] FLAG: --registry-qps="5" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151743 2577 flags.go:64] FLAG: --reserved-cpus="" Oct 30 20:59:32.157806 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151746 2577 flags.go:64] FLAG: --reserved-memory="" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151750 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151753 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151756 2577 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151759 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151764 2577 flags.go:64] FLAG: --runonce="false" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151767 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151770 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151773 2577 flags.go:64] FLAG: --seccomp-default="false" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151776 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151779 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151782 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151784 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151787 2577 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151790 2577 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151793 2577 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151796 2577 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151799 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151803 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151806 2577 flags.go:64] FLAG: --system-cgroups="" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151809 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151814 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151818 2577 flags.go:64] FLAG: --tls-cert-file="" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151822 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151826 2577 flags.go:64] FLAG: --tls-min-version="" Oct 30 20:59:32.158421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151829 2577 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 20:59:32.159004 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151832 2577 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 20:59:32.159004 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151835 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 20:59:32.159004 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151837 2577 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 20:59:32.159004 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151840 2577 flags.go:64] FLAG: --v="2" Oct 30 20:59:32.159004 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151844 2577 flags.go:64] FLAG: --version="false" Oct 30 20:59:32.159004 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151848 2577 flags.go:64] FLAG: --vmodule="" Oct 30 20:59:32.159004 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151852 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 20:59:32.159004 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.151856 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 20:59:32.159004 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.151990 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:59:32.159004 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.151993 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:59:32.159004 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.151997 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:59:32.159004 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152001 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:59:32.159004 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152003 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:59:32.159004 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152006 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:59:32.159004 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152009 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:59:32.159004 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152012 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:59:32.159004 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152014 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:59:32.159004 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152017 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:59:32.159004 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152019 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:59:32.159004 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152022 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:59:32.159004 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152024 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:59:32.159671 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152027 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:59:32.159671 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152041 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:59:32.159671 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152044 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:59:32.159671 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152046 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:59:32.159671 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152049 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:59:32.159671 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152052 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:59:32.159671 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152055 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:59:32.159671 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152059 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:59:32.159671 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152061 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:59:32.159671 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152064 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:59:32.159671 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152067 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:59:32.159671 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152069 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:59:32.159671 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152072 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:59:32.159671 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152074 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:59:32.159671 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152077 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:59:32.159671 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152079 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:59:32.159671 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152082 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:59:32.159671 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152085 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:59:32.159671 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152087 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:59:32.159671 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152090 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:59:32.160497 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152092 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:59:32.160497 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152095 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:59:32.160497 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152099 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:59:32.160497 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152102 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:59:32.160497 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152104 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:59:32.160497 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152107 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:59:32.160497 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152109 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:59:32.160497 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152112 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:59:32.160497 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152114 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:59:32.160497 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152116 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:59:32.160497 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152119 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:59:32.160497 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152121 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:59:32.160497 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152124 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:59:32.160497 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152127 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:59:32.160497 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152129 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:59:32.160497 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152132 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:59:32.160497 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152136 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:59:32.160497 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152139 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:59:32.160497 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152142 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:59:32.160497 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152162 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:59:32.161348 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152166 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:59:32.161348 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152169 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:59:32.161348 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152171 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:59:32.161348 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152174 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:59:32.161348 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152177 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:59:32.161348 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152179 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:59:32.161348 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152182 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:59:32.161348 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152184 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:59:32.161348 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152187 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:59:32.161348 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152190 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:59:32.161348 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152192 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:59:32.161348 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152195 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:59:32.161348 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152198 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:59:32.161348 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152200 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:59:32.161348 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152204 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:59:32.161348 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152207 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:59:32.161348 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152209 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:59:32.161348 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152212 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:59:32.161348 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152214 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:59:32.161348 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152217 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:59:32.162202 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152219 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:59:32.162202 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152221 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:59:32.162202 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152224 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:59:32.162202 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152226 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:59:32.162202 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152229 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:59:32.162202 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152232 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:59:32.162202 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152234 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:59:32.162202 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152236 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:59:32.162202 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152239 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:59:32.162202 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152242 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:59:32.162202 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152244 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:59:32.162202 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152248 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:59:32.162202 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152251 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:59:32.162202 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152253 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:59:32.162202 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152256 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:59:32.162202 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152258 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:59:32.162202 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152261 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:59:32.162202 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152263 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:59:32.162202 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152266 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:59:32.162202 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152268 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:59:32.163021 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152271 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:59:32.163021 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152273 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:59:32.163021 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152275 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:59:32.163021 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152278 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:59:32.163021 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152281 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:59:32.163021 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152283 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:59:32.163021 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152287 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:59:32.163021 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152289 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:59:32.163021 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152293 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:59:32.163021 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152297 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:59:32.163021 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152300 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:59:32.163021 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152302 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:59:32.163021 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.152305 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:59:32.163021 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.152330 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:59:32.163021 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153439 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:59:32.163670 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153565 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:59:32.163670 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153569 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:59:32.163670 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153572 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:59:32.163670 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153575 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:59:32.163670 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153578 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:59:32.163670 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153581 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:59:32.163670 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153583 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:59:32.163670 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153586 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:59:32.163670 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153588 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:59:32.163670 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153594 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:59:32.163670 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153599 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:59:32.163670 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153602 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:59:32.163670 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153605 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:59:32.163670 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153608 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:59:32.163670 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153610 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:59:32.163670 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153613 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:59:32.163670 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153616 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:59:32.163670 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153619 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:59:32.164185 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153622 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:59:32.164185 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153625 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:59:32.164185 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153627 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:59:32.164185 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153630 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:59:32.164185 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153633 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:59:32.164185 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153635 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:59:32.164185 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153638 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:59:32.164185 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153640 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:59:32.164185 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153643 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:59:32.164185 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153645 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:59:32.164185 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153648 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:59:32.164185 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153653 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:59:32.164185 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153655 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:59:32.164185 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153658 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:59:32.164185 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153662 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:59:32.164185 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153665 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:59:32.164185 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153668 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:59:32.164185 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153671 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:59:32.164185 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153673 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:59:32.164753 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153676 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:59:32.164753 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153680 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:59:32.164753 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153683 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:59:32.164753 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153686 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:59:32.164753 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153688 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:59:32.164753 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153690 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:59:32.164753 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153693 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:59:32.164753 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153696 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:59:32.164753 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153698 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:59:32.164753 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153700 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:59:32.164753 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153703 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:59:32.164753 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153705 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:59:32.164753 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153708 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:59:32.164753 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153711 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:59:32.164753 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153713 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:59:32.164753 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153716 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:59:32.164753 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153718 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:59:32.164753 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153721 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:59:32.164753 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153723 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:59:32.164753 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153726 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:59:32.165604 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153728 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:59:32.165604 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153731 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:59:32.165604 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153733 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:59:32.165604 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153736 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:59:32.165604 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153738 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:59:32.165604 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153742 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:59:32.165604 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153746 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:59:32.165604 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153753 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:59:32.165604 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153756 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:59:32.165604 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153759 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:59:32.165604 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153761 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:59:32.165604 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153764 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:59:32.165604 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153767 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:59:32.165604 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153769 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:59:32.165604 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153772 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:59:32.165604 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153774 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:59:32.165604 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153777 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:59:32.165604 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153779 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:59:32.165604 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153782 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:59:32.165604 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153784 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:59:32.166244 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153787 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:59:32.166244 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153789 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:59:32.166244 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153792 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:59:32.166244 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153794 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:59:32.166244 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153796 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:59:32.166244 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153799 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:59:32.166244 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153801 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:59:32.166244 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153804 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:59:32.166244 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153806 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:59:32.166244 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153809 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:59:32.166244 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153812 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:59:32.166244 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153814 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:59:32.166244 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153817 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:59:32.166244 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153819 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:59:32.166244 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153822 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:59:32.166244 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153825 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:59:32.166244 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153827 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:59:32.166244 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153830 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:59:32.166244 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153832 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:59:32.166244 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153838 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:59:32.166739 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153841 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:59:32.166739 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153844 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:59:32.166739 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153854 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:59:32.166739 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153858 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:59:32.166739 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153861 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:59:32.166739 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153864 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:59:32.166739 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153866 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:59:32.166739 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.153869 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:59:32.166739 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.161685 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 20:59:32.166739 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.161705 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 20:59:32.166739 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.161727 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 20:59:32.166739 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.161735 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 20:59:32.166739 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161838 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:59:32.166739 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161847 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:59:32.166739 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161861 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:59:32.166739 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161866 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:59:32.166739 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161870 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:59:32.166739 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161875 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:59:32.166739 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161880 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:59:32.167297 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161884 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:59:32.167297 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161888 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:59:32.167297 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161892 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:59:32.167297 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161896 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:59:32.167297 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161900 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:59:32.167297 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161904 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:59:32.167297 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161909 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:59:32.167297 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161913 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:59:32.167297 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161917 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:59:32.167297 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161921 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:59:32.167297 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161925 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:59:32.167297 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161930 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:59:32.167297 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161934 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:59:32.167297 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161938 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:59:32.167297 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161942 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:59:32.167297 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161946 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:59:32.167297 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161950 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:59:32.167297 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161955 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:59:32.167297 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161959 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:59:32.167734 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161963 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:59:32.167734 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161967 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:59:32.167734 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161971 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:59:32.167734 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161977 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:59:32.167734 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161982 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:59:32.167734 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161985 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:59:32.167734 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161990 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:59:32.167734 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161994 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:59:32.167734 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.161998 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:59:32.167734 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162002 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:59:32.167734 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162006 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:59:32.167734 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162010 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:59:32.167734 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162014 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:59:32.167734 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162018 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:59:32.167734 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162023 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:59:32.167734 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162027 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:59:32.167734 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162044 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:59:32.167734 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162049 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:59:32.167734 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162053 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:59:32.167734 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162057 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:59:32.168260 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162061 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:59:32.168260 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162065 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:59:32.168260 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162069 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:59:32.168260 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162073 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:59:32.168260 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162078 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:59:32.168260 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162084 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:59:32.168260 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162090 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:59:32.168260 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162096 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:59:32.168260 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162102 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:59:32.168260 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162107 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:59:32.168260 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162112 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:59:32.168260 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162116 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:59:32.168260 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162120 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:59:32.168260 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162124 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:59:32.168260 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162129 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:59:32.168260 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162134 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:59:32.168260 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162140 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:59:32.168260 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162144 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:59:32.168260 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162148 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:59:32.168687 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162152 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:59:32.168687 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162157 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:59:32.168687 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162161 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:59:32.168687 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162165 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:59:32.168687 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162170 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:59:32.168687 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162174 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:59:32.168687 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162178 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:59:32.168687 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162182 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:59:32.168687 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162187 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:59:32.168687 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162191 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:59:32.168687 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162195 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:59:32.168687 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162199 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:59:32.168687 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162203 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:59:32.168687 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162207 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:59:32.168687 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162210 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:59:32.168687 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162214 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:59:32.168687 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162218 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:59:32.168687 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162222 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:59:32.168687 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162226 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:59:32.168687 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162231 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:59:32.169314 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162235 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:59:32.169314 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162240 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:59:32.169314 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162245 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:59:32.169314 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162249 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:59:32.169314 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162253 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:59:32.169314 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162257 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:59:32.169314 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162261 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:59:32.169314 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162267 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:59:32.169314 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162271 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:59:32.169314 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162275 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:59:32.169314 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162280 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:59:32.169314 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162285 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:59:32.169314 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162290 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:59:32.169314 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162294 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:59:32.169314 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162298 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:59:32.169314 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162303 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:59:32.169314 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162307 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:59:32.169314 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162311 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:59:32.169314 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162315 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:59:32.169745 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162319 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:59:32.169745 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162323 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:59:32.169745 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.162364 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:59:32.169745 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162585 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:59:32.169745 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162594 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:59:32.169745 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162598 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:59:32.169745 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162602 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:59:32.169745 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162606 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:59:32.169745 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162611 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:59:32.169745 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162615 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:59:32.169745 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162619 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:59:32.169745 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162623 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:59:32.169745 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162627 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:59:32.169745 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162631 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:59:32.169745 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162635 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:59:32.169745 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162639 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:59:32.170360 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162643 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:59:32.170360 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162647 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:59:32.170360 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162651 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:59:32.170360 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162655 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:59:32.170360 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162660 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:59:32.170360 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162667 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:59:32.170360 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162674 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:59:32.170360 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162678 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:59:32.170360 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162683 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:59:32.170360 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162687 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:59:32.170360 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162692 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:59:32.170360 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162695 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:59:32.170360 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162699 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:59:32.170360 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162704 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:59:32.170360 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162708 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:59:32.170360 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162712 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:59:32.170360 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162716 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:59:32.170360 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162720 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:59:32.170360 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162725 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:59:32.171121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162729 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:59:32.171121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162734 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:59:32.171121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162738 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:59:32.171121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162742 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:59:32.171121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162746 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:59:32.171121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162750 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:59:32.171121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162755 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:59:32.171121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162759 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:59:32.171121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162763 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:59:32.171121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162767 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:59:32.171121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162771 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:59:32.171121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162775 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:59:32.171121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162779 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:59:32.171121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162783 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:59:32.171121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162787 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:59:32.171121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162791 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:59:32.171121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162795 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:59:32.171121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162800 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:59:32.171121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162805 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:59:32.171648 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162809 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:59:32.171648 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162813 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:59:32.171648 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162817 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:59:32.171648 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162822 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:59:32.171648 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162825 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:59:32.171648 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162830 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:59:32.171648 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162834 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:59:32.171648 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162838 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:59:32.171648 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162842 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:59:32.171648 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162846 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:59:32.171648 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162850 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:59:32.171648 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162854 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:59:32.171648 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162858 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:59:32.171648 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162862 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:59:32.171648 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162869 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:59:32.171648 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162876 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:59:32.171648 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162882 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:59:32.171648 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162887 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:59:32.171648 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162891 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:59:32.172121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162904 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:59:32.172121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162910 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:59:32.172121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162915 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:59:32.172121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162919 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:59:32.172121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162923 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:59:32.172121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162927 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:59:32.172121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162931 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:59:32.172121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162935 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:59:32.172121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162940 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:59:32.172121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162944 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:59:32.172121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162948 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:59:32.172121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162953 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:59:32.172121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162957 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:59:32.172121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162962 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:59:32.172121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162966 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:59:32.172121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162970 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:59:32.172121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162974 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:59:32.172121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162979 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:59:32.172121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162983 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:59:32.172121 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162987 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:59:32.172902 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162992 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:59:32.172902 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.162996 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:59:32.172902 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163000 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:59:32.172902 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163004 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:59:32.172902 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163008 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:59:32.172902 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163012 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:59:32.172902 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163017 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:59:32.172902 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163021 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:59:32.172902 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163025 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:59:32.172902 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163044 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:59:32.172902 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163048 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:59:32.172902 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163052 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:59:32.172902 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163056 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:59:32.172902 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163060 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:59:32.172902 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163065 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:59:32.172902 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163069 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:59:32.173394 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.163110 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 20:59:32.173394 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163161 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 20:59:32.173394 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163170 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 20:59:32.173394 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163175 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 20:59:32.173394 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163181 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 20:59:32.173394 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163186 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 20:59:32.173394 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163191 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 20:59:32.173394 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163195 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 20:59:32.173394 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163200 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 20:59:32.173394 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163204 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 20:59:32.173394 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163208 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 20:59:32.173394 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163212 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 20:59:32.173394 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163216 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 20:59:32.173394 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163220 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 20:59:32.173394 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163225 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 20:59:32.173394 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163230 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 20:59:32.173774 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163234 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 20:59:32.173774 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163238 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 20:59:32.173774 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163242 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 20:59:32.173774 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163247 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 20:59:32.173774 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163251 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 20:59:32.173774 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163255 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 20:59:32.173774 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163259 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 20:59:32.173774 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163262 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 20:59:32.173774 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163267 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 20:59:32.173774 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163271 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 20:59:32.173774 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163275 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 20:59:32.173774 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163280 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 20:59:32.173774 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163286 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 20:59:32.173774 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163291 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 20:59:32.173774 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163295 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 20:59:32.173774 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163299 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 20:59:32.173774 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163303 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 20:59:32.173774 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163307 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 20:59:32.173774 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163312 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 20:59:32.174354 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163316 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 20:59:32.174354 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163321 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 20:59:32.174354 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163325 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 20:59:32.174354 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163329 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 20:59:32.174354 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163333 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 20:59:32.174354 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163338 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 20:59:32.174354 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163342 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 20:59:32.174354 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163346 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 20:59:32.174354 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163350 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 20:59:32.174354 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163354 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 20:59:32.174354 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163358 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 20:59:32.174354 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163362 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 20:59:32.174354 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163367 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 20:59:32.174354 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163371 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 20:59:32.174354 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163375 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 20:59:32.174354 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163379 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 20:59:32.174354 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163383 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 20:59:32.174354 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163387 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 20:59:32.174354 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163391 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 20:59:32.174789 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163395 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 20:59:32.174789 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163399 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 20:59:32.174789 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163404 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 20:59:32.174789 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163408 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 20:59:32.174789 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163412 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 20:59:32.174789 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163416 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 20:59:32.174789 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163420 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 20:59:32.174789 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163424 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 20:59:32.174789 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163428 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 20:59:32.174789 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163432 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 20:59:32.174789 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163436 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 20:59:32.174789 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163440 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 20:59:32.174789 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163444 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 20:59:32.174789 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163448 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 20:59:32.174789 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163452 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 20:59:32.174789 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163456 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 20:59:32.174789 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163460 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 20:59:32.174789 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163465 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 20:59:32.174789 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163470 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 20:59:32.175272 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163474 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 20:59:32.175272 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163481 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 20:59:32.175272 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163486 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 20:59:32.175272 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163492 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 20:59:32.175272 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163496 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 20:59:32.175272 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163501 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 20:59:32.175272 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163508 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 20:59:32.175272 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163515 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 20:59:32.175272 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163519 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 20:59:32.175272 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163524 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 20:59:32.175272 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163528 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 20:59:32.175272 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163532 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 20:59:32.175272 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163536 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 20:59:32.175272 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163540 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 20:59:32.175272 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163544 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 20:59:32.175272 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163549 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 20:59:32.175272 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163553 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 20:59:32.175272 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163557 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 20:59:32.175272 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163561 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 20:59:32.175705 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163565 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 20:59:32.175705 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163570 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 20:59:32.175705 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163574 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 20:59:32.175705 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163578 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 20:59:32.175705 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163582 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 20:59:32.175705 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163586 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 20:59:32.175705 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163589 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 20:59:32.175705 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163594 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 20:59:32.175705 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163599 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 20:59:32.175705 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163603 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 20:59:32.175705 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163607 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 20:59:32.175705 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163611 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 20:59:32.175705 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163615 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 20:59:32.175705 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163619 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 20:59:32.175705 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:32.163625 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 20:59:32.175705 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.163704 2577 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 20:59:32.175705 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.166729 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 20:59:32.175705 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.167891 2577 server.go:1014] "Starting client certificate rotation" Oct 30 20:59:32.175705 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.167987 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 20:59:32.176153 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.168027 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 20:59:32.176153 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.170471 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:59:32.176153 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.173181 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 20:59:32.184720 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.184561 2577 log.go:25] "Validated CRI v1 runtime API" Oct 30 20:59:32.190691 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.190668 2577 log.go:25] "Validated CRI v1 image API" Oct 30 20:59:32.191614 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.191584 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 20:59:32.194424 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.194402 2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a2e7b3ab-3d6d-4b22-9176-ef965c843830:/dev/nvme0n1p3 b79046f5-774a-4003-b682-b478f32876d5:/dev/nvme0n1p4] Oct 30 20:59:32.194496 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.194424 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 20:59:32.199605 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.199487 2577 manager.go:217] Machine: {Timestamp:2026-10-30 20:59:32.19806056 +0000 UTC m=+0.314991750 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3198918 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22c8a44d470962e298a845cc7aa09c SystemUUID:ec22c8a4-4d47-0962-e298-a845cc7aa09c BootID:28d25c46-75df-4996-9cc7-b0820c83cb76 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ee:a5:d9:ff Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ee:a5:d9:ff Speed:0 Mtu:9001} {Name:ovs-system MacAddress:56:d4:95:7b:80:84 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 20:59:32.199605 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.199598 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 20:59:32.199761 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.199745 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 20:59:32.203641 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.203613 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 20:59:32.203812 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.203643 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-3.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 20:59:32.204519 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.204508 2577 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 20:59:32.204555 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.204521 2577 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 20:59:32.204555 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.204547 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:59:32.204612 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.204560 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 20:59:32.205847 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.205837 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 20:59:32.210640 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.210627 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 20:59:32.210770 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.210760 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 20:59:32.211689 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.211678 2577 kubelet.go:493] "Attempting to sync node with API server" Oct 30 20:59:32.211735 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.211695 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 20:59:32.211735 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.211716 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 20:59:32.211735 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.211730 2577 kubelet.go:405] "Adding apiserver pod source" Oct 30 20:59:32.211844 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.211741 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 20:59:32.212210 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.212197 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:59:32.214731 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.214713 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 20:59:32.221239 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.221224 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 20:59:32.222168 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.222156 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 20:59:32.222210 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.222191 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 20:59:32.222352 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.222342 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 20:59:32.222390 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.222357 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 20:59:32.222390 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.222363 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 20:59:32.222390 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.222368 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 20:59:32.222390 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.222374 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 20:59:32.222390 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.222379 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 20:59:32.223078 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.223070 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 20:59:32.223113 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.223079 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 20:59:32.223113 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.223087 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 20:59:32.223113 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.223092 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 20:59:32.223113 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.223100 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 20:59:32.223113 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.223108 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 20:59:32.224644 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.224636 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 20:59:32.224644 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.224645 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 20:59:32.228406 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.228394 2577 server.go:1263] "Started kubelet" Oct 30 20:59:32.228499 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.228450 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 20:59:32.228535 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.228488 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 20:59:32.228570 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.228557 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 20:59:32.229137 ip-10-0-142-3 systemd[1]: Started Kubernetes Kubelet. Oct 30 20:59:32.229984 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.229964 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 20:59:32.231748 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.231312 2577 server.go:317] "Adding debug handlers to kubelet server" Oct 30 20:59:32.235616 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:32.235600 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 20:59:32.237782 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.237339 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 20:59:32.238649 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.238636 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 20:59:32.238746 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:32.238723 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-3.ec2.internal\" not found" Oct 30 20:59:32.239555 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.239537 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 20:59:32.239629 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.239480 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 20:59:32.239629 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.239589 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 20:59:32.239724 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.239702 2577 reconstruct.go:97] "Volume reconstruction finished" Oct 30 20:59:32.239724 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.239710 2577 reconciler.go:29] "Reconciler: start to sync state" Oct 30 20:59:32.241891 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.241873 2577 factory.go:55] Registering systemd factory Oct 30 20:59:32.241891 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.241890 2577 factory.go:223] Registration of the systemd container factory successfully Oct 30 20:59:32.242324 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.242312 2577 factory.go:153] Registering CRI-O factory Oct 30 20:59:32.242324 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.242324 2577 factory.go:223] Registration of the crio container factory successfully Oct 30 20:59:32.242389 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.242375 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 20:59:32.242426 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.242393 2577 factory.go:103] Registering Raw factory Oct 30 20:59:32.242426 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.242409 2577 manager.go:1199] Started watching for new ooms in manager Oct 30 20:59:32.243526 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.243425 2577 manager.go:319] Starting recovery of all containers Oct 30 20:59:32.252677 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.252660 2577 manager.go:324] Recovery completed Oct 30 20:59:32.256619 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.256604 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:59:32.258461 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.258441 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-3.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 20:59:32.258572 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:32.258445 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-3.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 20:59:32.259271 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.259242 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasSufficientMemory" Oct 30 20:59:32.259331 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.259282 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 20:59:32.259331 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.259296 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasSufficientPID" Oct 30 20:59:32.259669 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:32.258435 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-3.ec2.internal.18e36a50e0ffa2f7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-3.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-3.ec2.internal,},FirstTimestamp:2026-10-30 20:59:32.228367095 +0000 UTC m=+0.345298284,LastTimestamp:2026-10-30 20:59:32.228367095 +0000 UTC m=+0.345298284,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-3.ec2.internal,}" Oct 30 20:59:32.259911 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.259898 2577 cpu_manager.go:226] "Starting" policy="none" Oct 30 20:59:32.259911 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.259909 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 20:59:32.260013 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.259924 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 20:59:32.260013 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.259941 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:59:32.261694 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.261677 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasSufficientMemory" Oct 30 20:59:32.261758 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.261709 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 20:59:32.261758 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.261722 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasSufficientPID" Oct 30 20:59:32.262313 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.262301 2577 policy_none.go:50] "Start" Oct 30 20:59:32.262345 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.262316 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 20:59:32.262345 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.262326 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 20:59:32.265921 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.265909 2577 policy_none.go:44] "Start" Oct 30 20:59:32.268641 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.268619 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 20:59:32.276697 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:32.276666 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 20:59:32.276789 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:32.276771 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-3.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 20:59:32.276789 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:32.276713 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-3.ec2.internal.18e36a50e2d73023 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-3.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-3.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-3.ec2.internal,},FirstTimestamp:2026-10-30 20:59:32.259270691 +0000 UTC m=+0.376201884,LastTimestamp:2026-10-30 20:59:32.259270691 +0000 UTC m=+0.376201884,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-3.ec2.internal,}" Oct 30 20:59:32.276910 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:32.276883 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 20:59:32.292073 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.290657 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wcmfl" Oct 30 20:59:32.296276 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.296259 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wcmfl" Oct 30 20:59:32.299073 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:32.298965 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-3.ec2.internal.18e36a50e2d776f9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-3.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-3.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-3.ec2.internal,},FirstTimestamp:2026-10-30 20:59:32.259288825 +0000 UTC m=+0.376220024,LastTimestamp:2026-10-30 20:59:32.259288825 +0000 UTC m=+0.376220024,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-3.ec2.internal,}" Oct 30 20:59:32.301646 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.301610 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 20:59:32.304196 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.304182 2577 manager.go:341] "Starting Device Plugin manager" Oct 30 20:59:32.304890 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:32.304877 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 20:59:32.304923 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.304892 2577 server.go:87] "Starting device plugin registration server" Oct 30 20:59:32.305152 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.305141 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 20:59:32.305192 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.305155 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 20:59:32.305290 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.305278 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 20:59:32.305368 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.305352 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 20:59:32.305368 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.305364 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 20:59:32.305811 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:32.305793 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 20:59:32.305877 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:32.305841 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-3.ec2.internal\" not found" Oct 30 20:59:32.364872 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.364842 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 20:59:32.366835 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.366013 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 20:59:32.366835 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.366047 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 20:59:32.366835 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.366069 2577 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 20:59:32.366835 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:32.366100 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 20:59:32.381319 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.381296 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:59:32.406177 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.406125 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:59:32.407088 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.407071 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasSufficientMemory" Oct 30 20:59:32.407179 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.407100 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 20:59:32.407179 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.407110 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasSufficientPID" Oct 30 20:59:32.407179 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.407141 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-3.ec2.internal" Oct 30 20:59:32.427661 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.427641 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-3.ec2.internal" Oct 30 20:59:32.427740 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:32.427665 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-3.ec2.internal\": node \"ip-10-0-142-3.ec2.internal\" not found" Oct 30 20:59:32.466969 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.466933 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-3.ec2.internal"] Oct 30 20:59:32.467086 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.467000 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:59:32.467173 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:32.467145 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-3.ec2.internal\" not found" Oct 30 20:59:32.467840 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.467826 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasSufficientMemory" Oct 30 20:59:32.467902 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.467851 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 20:59:32.467902 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.467865 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasSufficientPID" Oct 30 20:59:32.470075 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.470063 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:59:32.470210 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.470195 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" Oct 30 20:59:32.470246 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.470225 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:59:32.471894 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.471878 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasSufficientMemory" Oct 30 20:59:32.471968 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.471902 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 20:59:32.471968 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.471911 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasSufficientPID" Oct 30 20:59:32.471968 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.471884 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasSufficientMemory" Oct 30 20:59:32.472073 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.471973 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 20:59:32.472073 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.471982 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasSufficientPID" Oct 30 20:59:32.474289 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.474274 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-3.ec2.internal" Oct 30 20:59:32.474377 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.474305 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:59:32.476864 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.476849 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasSufficientMemory" Oct 30 20:59:32.476935 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.476882 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 20:59:32.476935 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.476897 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasSufficientPID" Oct 30 20:59:32.505737 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:32.505718 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-3.ec2.internal\" not found" node="ip-10-0-142-3.ec2.internal" Oct 30 20:59:32.508355 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:32.508340 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-3.ec2.internal\" not found" node="ip-10-0-142-3.ec2.internal" Oct 30 20:59:32.541934 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.541909 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/86fd282564aec729bf61e75aa114744a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal\" (UID: \"86fd282564aec729bf61e75aa114744a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" Oct 30 20:59:32.541934 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.541937 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86fd282564aec729bf61e75aa114744a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal\" (UID: \"86fd282564aec729bf61e75aa114744a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" Oct 30 20:59:32.542096 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.541952 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10ed65d1de0a53fd4cc159ce5efb836d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-3.ec2.internal\" (UID: \"10ed65d1de0a53fd4cc159ce5efb836d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-3.ec2.internal" Oct 30 20:59:32.567793 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:32.567753 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-3.ec2.internal\" not found" Oct 30 20:59:32.642902 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.642873 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/86fd282564aec729bf61e75aa114744a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal\" (UID: \"86fd282564aec729bf61e75aa114744a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" Oct 30 20:59:32.642902 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.642902 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86fd282564aec729bf61e75aa114744a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal\" (UID: \"86fd282564aec729bf61e75aa114744a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" Oct 30 20:59:32.643120 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.642920 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10ed65d1de0a53fd4cc159ce5efb836d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-3.ec2.internal\" (UID: \"10ed65d1de0a53fd4cc159ce5efb836d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-3.ec2.internal" Oct 30 20:59:32.643120 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.642982 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10ed65d1de0a53fd4cc159ce5efb836d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-3.ec2.internal\" (UID: \"10ed65d1de0a53fd4cc159ce5efb836d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-3.ec2.internal" Oct 30 20:59:32.643120 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.642988 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/86fd282564aec729bf61e75aa114744a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal\" (UID: \"86fd282564aec729bf61e75aa114744a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" Oct 30 20:59:32.643120 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.642986 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86fd282564aec729bf61e75aa114744a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal\" (UID: \"86fd282564aec729bf61e75aa114744a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" Oct 30 20:59:32.668396 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:32.668334 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-3.ec2.internal\" not found" Oct 30 20:59:32.768567 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:32.768541 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-3.ec2.internal\" not found" Oct 30 20:59:32.808756 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.808720 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" Oct 30 20:59:32.811874 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:32.811860 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-3.ec2.internal" Oct 30 20:59:32.869566 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:32.869411 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-3.ec2.internal\" not found" Oct 30 20:59:32.969913 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:32.969858 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-3.ec2.internal\" not found" Oct 30 20:59:33.070410 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:33.070384 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-3.ec2.internal\" not found" Oct 30 20:59:33.167959 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:33.167932 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 20:59:33.168376 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:33.168061 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 20:59:33.171087 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:33.171067 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-3.ec2.internal\" not found" Oct 30 20:59:33.272510 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:33.272144 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-3.ec2.internal\" not found" Oct 30 20:59:33.302756 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:33.302698 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:54:32 +0000 UTC" deadline="2028-06-11 01:04:27.903398121 +0000 UTC" Oct 30 20:59:33.302756 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:33.302748 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14140h4m54.600655998s" Oct 30 20:59:33.331562 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:33.331533 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:59:33.372846 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:33.372819 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-3.ec2.internal\" not found" Oct 30 20:59:33.433348 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:33.433317 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 20:59:33.434410 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:33.434389 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasSufficientMemory" Oct 30 20:59:33.434522 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:33.434429 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 20:59:33.434522 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:33.434444 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasSufficientPID" Oct 30 20:59:33.451515 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:33.451484 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86fd282564aec729bf61e75aa114744a.slice/crio-8ae91fed0dc7feaa07ff952a02edbe8cd026ec5af13f2e6e31a2a30d5a9015e7 WatchSource:0}: Error finding container 8ae91fed0dc7feaa07ff952a02edbe8cd026ec5af13f2e6e31a2a30d5a9015e7: Status 404 returned error can't find the container with id 8ae91fed0dc7feaa07ff952a02edbe8cd026ec5af13f2e6e31a2a30d5a9015e7 Oct 30 20:59:33.454293 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:33.454268 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10ed65d1de0a53fd4cc159ce5efb836d.slice/crio-9d5720d79d5bc05f502219a5b16458606d49b67752f9379196b789c214d91311 WatchSource:0}: Error finding container 9d5720d79d5bc05f502219a5b16458606d49b67752f9379196b789c214d91311: Status 404 returned error can't find the container with id 9d5720d79d5bc05f502219a5b16458606d49b67752f9379196b789c214d91311 Oct 30 20:59:33.456873 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:33.456853 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:59:33.473022 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:33.473005 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-3.ec2.internal\" not found" Oct 30 20:59:33.574593 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:33.574527 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-3.ec2.internal\" not found" Oct 30 20:59:33.674734 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:33.674697 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-3.ec2.internal\" not found" Oct 30 20:59:33.702661 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:33.702637 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:59:33.738815 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:33.738787 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" Oct 30 20:59:33.766476 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:33.766457 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:59:33.768200 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:33.768174 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-3.ec2.internal" Oct 30 20:59:33.789780 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:33.789764 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 20:59:34.211094 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.211037 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:59:34.213107 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.213083 2577 apiserver.go:52] "Watching apiserver" Oct 30 20:59:34.238878 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.238848 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 20:59:34.243270 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.243211 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 20:59:34.244013 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.243991 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-rkcv2","openshift-multus/multus-ggbwl","openshift-dns/node-resolver-54qxj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal","openshift-ovn-kubernetes/ovnkube-node-vtgj9","kube-system/konnectivity-agent-mgsr5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87","openshift-multus/network-metrics-daemon-7d6gs","kube-system/kube-apiserver-proxy-ip-10-0-142-3.ec2.internal","openshift-image-registry/node-ca-cnj4q","openshift-multus/multus-additional-cni-plugins-9gv9s","openshift-network-diagnostics/network-check-target-mfgjb","openshift-network-operator/iptables-alerter-ww5pw","kube-system/global-pull-secret-syncer-m9dm5"] Oct 30 20:59:34.246782 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.246758 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cnj4q" Oct 30 20:59:34.249362 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.249338 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ww5pw" Oct 30 20:59:34.251455 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.251435 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-54qxj" Oct 30 20:59:34.253653 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.253635 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:34.253749 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:34.253715 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9dm5" podUID="5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e" Oct 30 20:59:34.255930 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.255913 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.259033 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.259010 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mgsr5" Oct 30 20:59:34.261721 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.261702 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:34.261817 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:34.261770 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7d6gs" podUID="f6337834-db43-4b8e-a955-9691f72e0f19" Oct 30 20:59:34.262504 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.262480 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 20:59:34.262504 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.262490 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 20:59:34.263001 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.262811 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 20:59:34.263001 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.262968 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-79n9v\"" Oct 30 20:59:34.264502 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.264480 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:59:34.264760 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.264739 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 20:59:34.264850 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.264836 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.265529 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.265513 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j8v8x\"" Oct 30 20:59:34.265581 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.265571 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 20:59:34.266711 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.266591 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 20:59:34.266711 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.266649 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tnqjk\"" Oct 30 20:59:34.266711 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.266680 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 20:59:34.267547 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.267137 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.269932 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.269621 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.272239 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.272217 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.274775 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.274394 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 20:59:34.274775 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.274666 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 20:59:34.274775 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:34.274753 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfgjb" podUID="4e46662c-bfe6-45f8-8a23-89ea5fc92ae5" Oct 30 20:59:34.295305 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.295280 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 20:59:34.295812 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.295789 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mqhdd" Oct 30 20:59:34.343278 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.341081 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 20:59:34.354030 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354004 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-sys\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.354145 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354044 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03e64eb3-e7df-464d-9382-47cd0e570456-cni-binary-copy\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.354145 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354071 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:34.354251 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354149 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-registration-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.354251 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354192 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1362c712-7332-4a6e-a215-633d2f02acca-serviceca\") pod \"node-ca-cnj4q\" (UID: \"1362c712-7332-4a6e-a215-633d2f02acca\") " pod="openshift-image-registry/node-ca-cnj4q" Oct 30 20:59:34.354251 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354234 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b2ss\" (UniqueName: \"kubernetes.io/projected/a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31-kube-api-access-9b2ss\") pod \"node-resolver-54qxj\" (UID: \"a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31\") " pod="openshift-dns/node-resolver-54qxj" Oct 30 20:59:34.354452 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354276 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-run-openvswitch\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.354452 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354322 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e-agent-certs\") pod \"konnectivity-agent-mgsr5\" (UID: \"5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e\") " pod="kube-system/konnectivity-agent-mgsr5" Oct 30 20:59:34.354452 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354341 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-var-lib-cni-bin\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.354452 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354356 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.354452 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354403 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-cni-netd\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.354452 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354449 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03e64eb3-e7df-464d-9382-47cd0e570456-cnibin\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.354723 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354483 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-system-cni-dir\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.354723 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354508 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-etc-kubernetes\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.354723 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354533 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-log-socket\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.354723 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354594 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-ovnkube-config\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.354723 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354626 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-tuned\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.354723 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354650 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58ss\" (UniqueName: \"kubernetes.io/projected/1e20348d-9970-4de7-80be-84f29ef8ece6-kube-api-access-d58ss\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.354723 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354670 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x799c\" (UniqueName: \"kubernetes.io/projected/8bc258c0-5d93-4ddb-b49c-9a06167f6185-kube-api-access-x799c\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.354723 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354720 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-kubernetes\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.355056 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354757 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03e64eb3-e7df-464d-9382-47cd0e570456-system-cni-dir\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.355056 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354802 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs\") pod \"network-metrics-daemon-7d6gs\" (UID: \"f6337834-db43-4b8e-a955-9691f72e0f19\") " pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:34.355056 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354829 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bc258c0-5d93-4ddb-b49c-9a06167f6185-cni-binary-copy\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.355056 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354877 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-systemd-units\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.355056 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354898 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-cni-bin\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.355056 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354935 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67blk\" (UniqueName: \"kubernetes.io/projected/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-kube-api-access-67blk\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.355056 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354964 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-var-lib-kubelet\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.355056 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.354995 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-etc-openvswitch\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.355056 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355016 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-host\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.355056 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355057 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-run-netns\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.355472 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355104 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-sysctl-conf\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.355472 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355134 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xcm4\" (UniqueName: \"kubernetes.io/projected/03e64eb3-e7df-464d-9382-47cd0e570456-kube-api-access-9xcm4\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.355472 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355167 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e-konnectivity-ca\") pod \"konnectivity-agent-mgsr5\" (UID: \"5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e\") " pod="kube-system/konnectivity-agent-mgsr5" Oct 30 20:59:34.355472 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355215 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz6tt\" (UniqueName: \"kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt\") pod \"network-check-target-mfgjb\" (UID: \"4e46662c-bfe6-45f8-8a23-89ea5fc92ae5\") " pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 20:59:34.355472 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355247 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2tfx\" (UniqueName: \"kubernetes.io/projected/97872ff8-a1f5-4062-8202-41aa9d8a93bc-kube-api-access-k2tfx\") pod \"iptables-alerter-ww5pw\" (UID: \"97872ff8-a1f5-4062-8202-41aa9d8a93bc\") " pod="openshift-network-operator/iptables-alerter-ww5pw" Oct 30 20:59:34.355472 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355271 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-run-ovn\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.355472 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355295 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.355472 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355311 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-run\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.355472 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355326 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-run-k8s-cni-cncf-io\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.355472 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355370 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-var-lib-cni-multus\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.355472 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355398 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-slash\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.355472 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355415 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-systemd\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.355472 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355430 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-lib-modules\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.355472 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355445 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03e64eb3-e7df-464d-9382-47cd0e570456-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.355472 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355463 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03e64eb3-e7df-464d-9382-47cd0e570456-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.355969 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355481 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.355969 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355518 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-conf-dir\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.355969 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355534 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-node-log\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.355969 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355548 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-env-overrides\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.355969 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355562 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e20348d-9970-4de7-80be-84f29ef8ece6-tmp\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.355969 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355581 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/97872ff8-a1f5-4062-8202-41aa9d8a93bc-iptables-alerter-script\") pod \"iptables-alerter-ww5pw\" (UID: \"97872ff8-a1f5-4062-8202-41aa9d8a93bc\") " pod="openshift-network-operator/iptables-alerter-ww5pw" Oct 30 20:59:34.355969 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355595 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31-hosts-file\") pod \"node-resolver-54qxj\" (UID: \"a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31\") " pod="openshift-dns/node-resolver-54qxj" Oct 30 20:59:34.355969 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355610 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-sys-fs\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.355969 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355624 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-var-lib-kubelet\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.355969 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355644 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-run-multus-certs\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.355969 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355669 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-sysctl-d\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.355969 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355688 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03e64eb3-e7df-464d-9382-47cd0e570456-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.355969 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355707 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97872ff8-a1f5-4062-8202-41aa9d8a93bc-host-slash\") pod \"iptables-alerter-ww5pw\" (UID: \"97872ff8-a1f5-4062-8202-41aa9d8a93bc\") " pod="openshift-network-operator/iptables-alerter-ww5pw" Oct 30 20:59:34.355969 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355722 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g67v\" (UniqueName: \"kubernetes.io/projected/93f2c084-a1d2-4bda-8a41-c01829cc81ba-kube-api-access-6g67v\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.355969 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355736 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-cnibin\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.355969 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355751 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-hostroot\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.356513 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355812 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-daemon-config\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.356513 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355828 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1362c712-7332-4a6e-a215-633d2f02acca-host\") pod \"node-ca-cnj4q\" (UID: \"1362c712-7332-4a6e-a215-633d2f02acca\") " pod="openshift-image-registry/node-ca-cnj4q" Oct 30 20:59:34.356513 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355845 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-dbus\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:34.356513 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355870 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-socket-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.356513 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355893 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-run-netns\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.356513 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355918 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31-tmp-dir\") pod \"node-resolver-54qxj\" (UID: \"a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31\") " pod="openshift-dns/node-resolver-54qxj" Oct 30 20:59:34.356513 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355940 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7cv8\" (UniqueName: \"kubernetes.io/projected/f6337834-db43-4b8e-a955-9691f72e0f19-kube-api-access-f7cv8\") pod \"network-metrics-daemon-7d6gs\" (UID: \"f6337834-db43-4b8e-a955-9691f72e0f19\") " pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:34.356513 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355955 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-device-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.356513 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355970 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-run-systemd\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.356513 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.355984 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-modprobe-d\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.356513 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.356000 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-kubelet-config\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:34.356513 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.356016 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-etc-selinux\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.356513 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.356030 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-cni-dir\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.356513 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.356050 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-var-lib-openvswitch\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.356513 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.356065 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-ovn-node-metrics-cert\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.356513 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.356102 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-sysconfig\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.357004 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.356146 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhhrx\" (UniqueName: \"kubernetes.io/projected/1362c712-7332-4a6e-a215-633d2f02acca-kube-api-access-dhhrx\") pod \"node-ca-cnj4q\" (UID: \"1362c712-7332-4a6e-a215-633d2f02acca\") " pod="openshift-image-registry/node-ca-cnj4q" Oct 30 20:59:34.357004 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.356172 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-socket-dir-parent\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.357004 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.356199 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-kubelet\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.357004 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.356215 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-ovnkube-script-lib\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.372536 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.371796 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 20:59:34.372536 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.371847 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 20:59:34.372536 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.371868 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b8vzq\"" Oct 30 20:59:34.372536 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.371909 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 20:59:34.372536 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.371948 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 20:59:34.372536 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.371968 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 20:59:34.372536 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.372265 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 20:59:34.372536 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.372306 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4x2zw\"" Oct 30 20:59:34.372536 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.372309 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 20:59:34.372536 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.372348 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 20:59:34.372536 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.372368 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 20:59:34.372536 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.372385 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 20:59:34.372536 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.372419 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 20:59:34.372536 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.372354 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 20:59:34.372536 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.372503 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 20:59:34.373340 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.372939 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-3.ec2.internal" event={"ID":"10ed65d1de0a53fd4cc159ce5efb836d","Type":"ContainerStarted","Data":"9d5720d79d5bc05f502219a5b16458606d49b67752f9379196b789c214d91311"} Oct 30 20:59:34.373759 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.373589 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 20:59:34.373759 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.373652 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 20:59:34.373759 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.373668 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tzc2p\"" Oct 30 20:59:34.373759 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.373671 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 20:59:34.373759 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.373693 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gszwc\"" Oct 30 20:59:34.373759 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.373699 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 20:59:34.373759 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.373700 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f7r6m\"" Oct 30 20:59:34.373759 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.373712 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 20:59:34.373759 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.373740 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-hvtl2\"" Oct 30 20:59:34.374254 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.374143 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" event={"ID":"86fd282564aec729bf61e75aa114744a","Type":"ContainerStarted","Data":"8ae91fed0dc7feaa07ff952a02edbe8cd026ec5af13f2e6e31a2a30d5a9015e7"} Oct 30 20:59:34.427221 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.427193 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 20:59:34.457350 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457319 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-run-k8s-cni-cncf-io\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.457492 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457360 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-var-lib-cni-multus\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.457492 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457387 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-slash\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.457492 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457405 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-systemd\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.457492 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457405 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-run-k8s-cni-cncf-io\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.457492 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457421 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-lib-modules\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.457492 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457465 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03e64eb3-e7df-464d-9382-47cd0e570456-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.457492 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457480 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-slash\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.457735 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457496 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03e64eb3-e7df-464d-9382-47cd0e570456-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.457735 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457525 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.457735 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457532 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-var-lib-cni-multus\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.457735 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457549 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-conf-dir\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.457735 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457561 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-lib-modules\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.457735 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457574 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-node-log\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.457735 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457612 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-env-overrides\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.457735 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457627 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e20348d-9970-4de7-80be-84f29ef8ece6-tmp\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.457735 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457629 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.457735 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457652 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/97872ff8-a1f5-4062-8202-41aa9d8a93bc-iptables-alerter-script\") pod \"iptables-alerter-ww5pw\" (UID: \"97872ff8-a1f5-4062-8202-41aa9d8a93bc\") " pod="openshift-network-operator/iptables-alerter-ww5pw" Oct 30 20:59:34.457735 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457668 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-conf-dir\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.457735 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457672 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31-hosts-file\") pod \"node-resolver-54qxj\" (UID: \"a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31\") " pod="openshift-dns/node-resolver-54qxj" Oct 30 20:59:34.457735 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457704 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31-hosts-file\") pod \"node-resolver-54qxj\" (UID: \"a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31\") " pod="openshift-dns/node-resolver-54qxj" Oct 30 20:59:34.457735 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457704 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-sys-fs\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.457735 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457731 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-var-lib-kubelet\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.458334 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457747 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-systemd\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.458334 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457757 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-run-multus-certs\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.458334 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457789 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-sysctl-d\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.458334 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457813 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03e64eb3-e7df-464d-9382-47cd0e570456-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.458334 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457840 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97872ff8-a1f5-4062-8202-41aa9d8a93bc-host-slash\") pod \"iptables-alerter-ww5pw\" (UID: \"97872ff8-a1f5-4062-8202-41aa9d8a93bc\") " pod="openshift-network-operator/iptables-alerter-ww5pw" Oct 30 20:59:34.458334 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457866 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g67v\" (UniqueName: \"kubernetes.io/projected/93f2c084-a1d2-4bda-8a41-c01829cc81ba-kube-api-access-6g67v\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.458334 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457893 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-cnibin\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.458334 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457915 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-hostroot\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.458334 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457940 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-daemon-config\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.458334 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457963 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1362c712-7332-4a6e-a215-633d2f02acca-host\") pod \"node-ca-cnj4q\" (UID: \"1362c712-7332-4a6e-a215-633d2f02acca\") " pod="openshift-image-registry/node-ca-cnj4q" Oct 30 20:59:34.458334 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.457987 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-dbus\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:34.458334 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458012 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-socket-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.458334 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458031 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 20:59:34.458334 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458041 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-run-netns\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.458334 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458066 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31-tmp-dir\") pod \"node-resolver-54qxj\" (UID: \"a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31\") " pod="openshift-dns/node-resolver-54qxj" Oct 30 20:59:34.458334 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458108 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7cv8\" (UniqueName: \"kubernetes.io/projected/f6337834-db43-4b8e-a955-9691f72e0f19-kube-api-access-f7cv8\") pod \"network-metrics-daemon-7d6gs\" (UID: \"f6337834-db43-4b8e-a955-9691f72e0f19\") " pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:34.458334 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458136 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-device-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.458334 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458161 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-run-systemd\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.459250 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458185 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-modprobe-d\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.459250 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458247 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-kubelet-config\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:34.459250 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458279 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-etc-selinux\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.459250 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458282 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-hostroot\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.459250 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458309 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-cni-dir\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.459250 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458338 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-var-lib-openvswitch\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.459250 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458352 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-sys-fs\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.459250 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458364 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-ovn-node-metrics-cert\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.459250 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458419 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-var-lib-kubelet\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.459250 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458440 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-node-log\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.459250 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458452 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-sysconfig\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.459250 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458491 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhhrx\" (UniqueName: \"kubernetes.io/projected/1362c712-7332-4a6e-a215-633d2f02acca-kube-api-access-dhhrx\") pod \"node-ca-cnj4q\" (UID: \"1362c712-7332-4a6e-a215-633d2f02acca\") " pod="openshift-image-registry/node-ca-cnj4q" Oct 30 20:59:34.459250 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458497 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-sysconfig\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.459250 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458523 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-socket-dir-parent\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.459250 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458544 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-run-multus-certs\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.459250 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458555 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-kubelet\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.459250 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458581 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-ovnkube-script-lib\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.459250 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458614 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-sys\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.460124 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458639 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03e64eb3-e7df-464d-9382-47cd0e570456-cni-binary-copy\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.460124 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458666 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:34.460124 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458690 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-registration-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.460124 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458723 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1362c712-7332-4a6e-a215-633d2f02acca-serviceca\") pod \"node-ca-cnj4q\" (UID: \"1362c712-7332-4a6e-a215-633d2f02acca\") " pod="openshift-image-registry/node-ca-cnj4q" Oct 30 20:59:34.460124 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458722 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97872ff8-a1f5-4062-8202-41aa9d8a93bc-host-slash\") pod \"iptables-alerter-ww5pw\" (UID: \"97872ff8-a1f5-4062-8202-41aa9d8a93bc\") " pod="openshift-network-operator/iptables-alerter-ww5pw" Oct 30 20:59:34.460124 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458754 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03e64eb3-e7df-464d-9382-47cd0e570456-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.460124 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458842 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-sysctl-d\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.460124 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458886 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b2ss\" (UniqueName: \"kubernetes.io/projected/a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31-kube-api-access-9b2ss\") pod \"node-resolver-54qxj\" (UID: \"a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31\") " pod="openshift-dns/node-resolver-54qxj" Oct 30 20:59:34.460124 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458917 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-run-openvswitch\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.460124 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458944 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e-agent-certs\") pod \"konnectivity-agent-mgsr5\" (UID: \"5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e\") " pod="kube-system/konnectivity-agent-mgsr5" Oct 30 20:59:34.460124 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458971 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-var-lib-cni-bin\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.460124 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458977 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-socket-dir-parent\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.460124 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.458997 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.460124 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459007 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-kubelet\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.460124 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459025 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-cni-netd\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.460124 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459051 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03e64eb3-e7df-464d-9382-47cd0e570456-cnibin\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.460124 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459079 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-system-cni-dir\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.460955 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459128 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-etc-kubernetes\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.460955 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459153 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-log-socket\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.460955 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459186 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-ovnkube-config\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.460955 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459212 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-tuned\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.460955 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459237 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58ss\" (UniqueName: \"kubernetes.io/projected/1e20348d-9970-4de7-80be-84f29ef8ece6-kube-api-access-d58ss\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.460955 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459249 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-daemon-config\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.460955 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459261 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x799c\" (UniqueName: \"kubernetes.io/projected/8bc258c0-5d93-4ddb-b49c-9a06167f6185-kube-api-access-x799c\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.460955 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459286 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-kubernetes\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.460955 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459313 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1362c712-7332-4a6e-a215-633d2f02acca-host\") pod \"node-ca-cnj4q\" (UID: \"1362c712-7332-4a6e-a215-633d2f02acca\") " pod="openshift-image-registry/node-ca-cnj4q" Oct 30 20:59:34.460955 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459319 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03e64eb3-e7df-464d-9382-47cd0e570456-system-cni-dir\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.460955 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459348 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs\") pod \"network-metrics-daemon-7d6gs\" (UID: \"f6337834-db43-4b8e-a955-9691f72e0f19\") " pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:34.460955 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459377 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bc258c0-5d93-4ddb-b49c-9a06167f6185-cni-binary-copy\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.460955 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-systemd-units\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.460955 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459415 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-socket-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.460955 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459429 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-cni-bin\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.460955 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459457 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67blk\" (UniqueName: \"kubernetes.io/projected/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-kube-api-access-67blk\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.460955 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459463 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-run-netns\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.461801 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459482 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-var-lib-kubelet\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.461801 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459509 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-etc-openvswitch\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.461801 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459533 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-host\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.461801 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459551 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-env-overrides\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.461801 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459559 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-run-netns\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.461801 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459551 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03e64eb3-e7df-464d-9382-47cd0e570456-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.461801 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459587 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-sysctl-conf\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.461801 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459603 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-ovnkube-script-lib\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.461801 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459616 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xcm4\" (UniqueName: \"kubernetes.io/projected/03e64eb3-e7df-464d-9382-47cd0e570456-kube-api-access-9xcm4\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.461801 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:34.459625 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:59:34.461801 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459642 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-cnibin\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.461801 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459657 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e-konnectivity-ca\") pod \"konnectivity-agent-mgsr5\" (UID: \"5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e\") " pod="kube-system/konnectivity-agent-mgsr5" Oct 30 20:59:34.461801 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459686 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz6tt\" (UniqueName: \"kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt\") pod \"network-check-target-mfgjb\" (UID: \"4e46662c-bfe6-45f8-8a23-89ea5fc92ae5\") " pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 20:59:34.461801 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459698 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-kubernetes\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.461801 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459711 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2tfx\" (UniqueName: \"kubernetes.io/projected/97872ff8-a1f5-4062-8202-41aa9d8a93bc-kube-api-access-k2tfx\") pod \"iptables-alerter-ww5pw\" (UID: \"97872ff8-a1f5-4062-8202-41aa9d8a93bc\") " pod="openshift-network-operator/iptables-alerter-ww5pw" Oct 30 20:59:34.461801 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459712 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31-tmp-dir\") pod \"node-resolver-54qxj\" (UID: \"a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31\") " pod="openshift-dns/node-resolver-54qxj" Oct 30 20:59:34.461801 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459736 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-run-ovn\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.461801 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.462744 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459286 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/97872ff8-a1f5-4062-8202-41aa9d8a93bc-iptables-alerter-script\") pod \"iptables-alerter-ww5pw\" (UID: \"97872ff8-a1f5-4062-8202-41aa9d8a93bc\") " pod="openshift-network-operator/iptables-alerter-ww5pw" Oct 30 20:59:34.462744 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459783 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-run\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.462744 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459822 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-run\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.462744 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459829 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03e64eb3-e7df-464d-9382-47cd0e570456-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.462744 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459863 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-registration-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.462744 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459888 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-run-openvswitch\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.462744 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459884 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-run-systemd\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.462744 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459909 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-sys\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.462744 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459934 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03e64eb3-e7df-464d-9382-47cd0e570456-system-cni-dir\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.462744 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.459989 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-device-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.462744 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460093 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-dbus\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:34.462744 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460167 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-etc-selinux\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.462744 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:34.460199 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:59:34.462744 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460254 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1362c712-7332-4a6e-a215-633d2f02acca-serviceca\") pod \"node-ca-cnj4q\" (UID: \"1362c712-7332-4a6e-a215-633d2f02acca\") " pod="openshift-image-registry/node-ca-cnj4q" Oct 30 20:59:34.462744 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460265 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-modprobe-d\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.462744 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:34.460285 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs podName:f6337834-db43-4b8e-a955-9691f72e0f19 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:34.960244426 +0000 UTC m=+3.077175620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs") pod "network-metrics-daemon-7d6gs" (UID: "f6337834-db43-4b8e-a955-9691f72e0f19") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:59:34.462744 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460302 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-kubelet-config\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:34.463564 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:34.460349 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret podName:5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e nodeName:}" failed. No retries permitted until 2026-10-30 20:59:34.960326935 +0000 UTC m=+3.077258127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret") pod "global-pull-secret-syncer-m9dm5" (UID: "5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:59:34.463564 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460346 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03e64eb3-e7df-464d-9382-47cd0e570456-cni-binary-copy\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.463564 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460409 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-host\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.463564 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460413 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-run-ovn\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.463564 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460452 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.463564 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460467 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-systemd-units\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.463564 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460511 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-var-lib-openvswitch\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.463564 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460547 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-run-netns\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.463564 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460570 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-sysctl-conf\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.463564 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460585 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-cni-bin\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.463564 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460682 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bc258c0-5d93-4ddb-b49c-9a06167f6185-cni-binary-copy\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.463564 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460728 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-cni-dir\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.463564 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460758 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-system-cni-dir\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.463564 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460762 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-var-lib-cni-bin\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.463564 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460786 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.463564 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460819 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-var-lib-kubelet\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.463564 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460821 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-cni-netd\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.464311 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460845 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-etc-openvswitch\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.464311 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460846 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e-konnectivity-ca\") pod \"konnectivity-agent-mgsr5\" (UID: \"5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e\") " pod="kube-system/konnectivity-agent-mgsr5" Oct 30 20:59:34.464311 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460868 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03e64eb3-e7df-464d-9382-47cd0e570456-cnibin\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.464311 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.460920 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-etc-kubernetes\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.464311 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.461023 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-log-socket\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.464311 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.461129 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-ovnkube-config\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.464311 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.463020 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e20348d-9970-4de7-80be-84f29ef8ece6-tmp\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.464311 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.463339 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e-agent-certs\") pod \"konnectivity-agent-mgsr5\" (UID: \"5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e\") " pod="kube-system/konnectivity-agent-mgsr5" Oct 30 20:59:34.464311 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.464029 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-tuned\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.464772 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.464744 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-ovn-node-metrics-cert\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.486130 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:34.479426 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:59:34.486130 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:34.479454 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:59:34.486130 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:34.479468 2577 projected.go:196] Error preparing data for projected volume kube-api-access-dz6tt for pod openshift-network-diagnostics/network-check-target-mfgjb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:59:34.486130 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:34.479526 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt podName:4e46662c-bfe6-45f8-8a23-89ea5fc92ae5 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:34.979507568 +0000 UTC m=+3.096438746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dz6tt" (UniqueName: "kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt") pod "network-check-target-mfgjb" (UID: "4e46662c-bfe6-45f8-8a23-89ea5fc92ae5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:59:34.486130 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.479847 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhhrx\" (UniqueName: \"kubernetes.io/projected/1362c712-7332-4a6e-a215-633d2f02acca-kube-api-access-dhhrx\") pod \"node-ca-cnj4q\" (UID: \"1362c712-7332-4a6e-a215-633d2f02acca\") " pod="openshift-image-registry/node-ca-cnj4q" Oct 30 20:59:34.486130 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.480949 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7cv8\" (UniqueName: \"kubernetes.io/projected/f6337834-db43-4b8e-a955-9691f72e0f19-kube-api-access-f7cv8\") pod \"network-metrics-daemon-7d6gs\" (UID: \"f6337834-db43-4b8e-a955-9691f72e0f19\") " pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:34.486130 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.481927 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xcm4\" (UniqueName: \"kubernetes.io/projected/03e64eb3-e7df-464d-9382-47cd0e570456-kube-api-access-9xcm4\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.486130 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.482334 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2tfx\" (UniqueName: \"kubernetes.io/projected/97872ff8-a1f5-4062-8202-41aa9d8a93bc-kube-api-access-k2tfx\") pod \"iptables-alerter-ww5pw\" (UID: \"97872ff8-a1f5-4062-8202-41aa9d8a93bc\") " pod="openshift-network-operator/iptables-alerter-ww5pw" Oct 30 20:59:34.486130 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.483631 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x799c\" (UniqueName: \"kubernetes.io/projected/8bc258c0-5d93-4ddb-b49c-9a06167f6185-kube-api-access-x799c\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.486130 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.484648 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g67v\" (UniqueName: \"kubernetes.io/projected/93f2c084-a1d2-4bda-8a41-c01829cc81ba-kube-api-access-6g67v\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.486130 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.484787 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b2ss\" (UniqueName: \"kubernetes.io/projected/a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31-kube-api-access-9b2ss\") pod \"node-resolver-54qxj\" (UID: \"a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31\") " pod="openshift-dns/node-resolver-54qxj" Oct 30 20:59:34.486130 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.484827 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58ss\" (UniqueName: \"kubernetes.io/projected/1e20348d-9970-4de7-80be-84f29ef8ece6-kube-api-access-d58ss\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.486822 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.486428 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67blk\" (UniqueName: \"kubernetes.io/projected/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-kube-api-access-67blk\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.562712 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.562681 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cnj4q" Oct 30 20:59:34.573792 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.573763 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ww5pw" Oct 30 20:59:34.575232 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:34.575194 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1362c712_7332_4a6e_a215_633d2f02acca.slice/crio-bf089dbea2ba22737b1bb71e42206d667d7b745dcaa421f6d6abca101caa3ceb WatchSource:0}: Error finding container bf089dbea2ba22737b1bb71e42206d667d7b745dcaa421f6d6abca101caa3ceb: Status 404 returned error can't find the container with id bf089dbea2ba22737b1bb71e42206d667d7b745dcaa421f6d6abca101caa3ceb Oct 30 20:59:34.582761 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.582726 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-54qxj" Oct 30 20:59:34.584714 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:34.584676 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97872ff8_a1f5_4062_8202_41aa9d8a93bc.slice/crio-c9e921e24c64a15435106455241e0ff8bfd38a668102ecd4b56609cc4ed29f10 WatchSource:0}: Error finding container c9e921e24c64a15435106455241e0ff8bfd38a668102ecd4b56609cc4ed29f10: Status 404 returned error can't find the container with id c9e921e24c64a15435106455241e0ff8bfd38a668102ecd4b56609cc4ed29f10 Oct 30 20:59:34.593399 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:34.593369 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1d0e9c5_2d37_4adf_a4a8_b4da4fbaae31.slice/crio-4961a255689d4f8b3ca3f78ac06b9076bfbd9659ab98826173df297fdb40eda9 WatchSource:0}: Error finding container 4961a255689d4f8b3ca3f78ac06b9076bfbd9659ab98826173df297fdb40eda9: Status 404 returned error can't find the container with id 4961a255689d4f8b3ca3f78ac06b9076bfbd9659ab98826173df297fdb40eda9 Oct 30 20:59:34.608179 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.608152 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 20:59:34.616871 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.616851 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mgsr5" Oct 30 20:59:34.618563 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:34.618536 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03e64eb3_e7df_464d_9382_47cd0e570456.slice/crio-fcc5c3c5eb56c860c84b8d31c502139f42a747a5ad1b6e0105faf9366a948c44 WatchSource:0}: Error finding container fcc5c3c5eb56c860c84b8d31c502139f42a747a5ad1b6e0105faf9366a948c44: Status 404 returned error can't find the container with id fcc5c3c5eb56c860c84b8d31c502139f42a747a5ad1b6e0105faf9366a948c44 Oct 30 20:59:34.627578 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:34.627553 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5525d794_7ab8_4d4a_84fa_bfa14d9d3d8e.slice/crio-03dab2bd4f5ccd547d64bb7a5331444dfb48a537d0ebfa55136fc411976c629d WatchSource:0}: Error finding container 03dab2bd4f5ccd547d64bb7a5331444dfb48a537d0ebfa55136fc411976c629d: Status 404 returned error can't find the container with id 03dab2bd4f5ccd547d64bb7a5331444dfb48a537d0ebfa55136fc411976c629d Oct 30 20:59:34.634992 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.634970 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ggbwl" Oct 30 20:59:34.637419 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.637398 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:34.639484 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.639463 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 20:59:34.642674 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.642645 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 20:59:34.653324 ip-10-0-142-3 kubenswrapper[2577]: W1030 20:59:34.653297 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88f212cc_4eb8_4a49_9c13_a1d395ab43c6.slice/crio-9fd2d7bbece44d55d7b897ad7783d27cac24ed24ae77eeed57b1a3de942d3cb5 WatchSource:0}: Error finding container 9fd2d7bbece44d55d7b897ad7783d27cac24ed24ae77eeed57b1a3de942d3cb5: Status 404 returned error can't find the container with id 9fd2d7bbece44d55d7b897ad7783d27cac24ed24ae77eeed57b1a3de942d3cb5 Oct 30 20:59:34.963176 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.963145 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:34.963278 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:34.963189 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs\") pod \"network-metrics-daemon-7d6gs\" (UID: \"f6337834-db43-4b8e-a955-9691f72e0f19\") " pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:34.963351 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:34.963335 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:59:34.963408 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:34.963392 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs podName:f6337834-db43-4b8e-a955-9691f72e0f19 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:35.963373598 +0000 UTC m=+4.080304792 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs") pod "network-metrics-daemon-7d6gs" (UID: "f6337834-db43-4b8e-a955-9691f72e0f19") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:59:34.963764 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:34.963744 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:59:34.963854 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:34.963799 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret podName:5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e nodeName:}" failed. No retries permitted until 2026-10-30 20:59:35.963783931 +0000 UTC m=+4.080715125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret") pod "global-pull-secret-syncer-m9dm5" (UID: "5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:59:35.065019 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:35.064532 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz6tt\" (UniqueName: \"kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt\") pod \"network-check-target-mfgjb\" (UID: \"4e46662c-bfe6-45f8-8a23-89ea5fc92ae5\") " pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 20:59:35.065019 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:35.064711 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:59:35.065019 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:35.064732 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:59:35.065019 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:35.064744 2577 projected.go:196] Error preparing data for projected volume kube-api-access-dz6tt for pod openshift-network-diagnostics/network-check-target-mfgjb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:59:35.065019 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:35.064808 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt podName:4e46662c-bfe6-45f8-8a23-89ea5fc92ae5 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:36.06479011 +0000 UTC m=+4.181721297 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dz6tt" (UniqueName: "kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt") pod "network-check-target-mfgjb" (UID: "4e46662c-bfe6-45f8-8a23-89ea5fc92ae5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:59:35.299326 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:35.299200 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:54:34 +0000 UTC" deadline="2028-06-26 11:13:15.030759744 +0000 UTC" Oct 30 20:59:35.299326 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:35.299272 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14510h13m39.73149217s" Oct 30 20:59:35.387207 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:35.386390 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" event={"ID":"86fd282564aec729bf61e75aa114744a","Type":"ContainerStarted","Data":"3ed8e2409579bd986770f5c33540d4899bae49deadbd613baab5f08db05729cd"} Oct 30 20:59:35.389016 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:35.388937 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" event={"ID":"1e20348d-9970-4de7-80be-84f29ef8ece6","Type":"ContainerStarted","Data":"6e0e41eaec7a06f51e0c0e48a5426b59954aba345f3c9d7158e57db245ef1d63"} Oct 30 20:59:35.392172 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:35.392062 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" event={"ID":"88f212cc-4eb8-4a49-9c13-a1d395ab43c6","Type":"ContainerStarted","Data":"9fd2d7bbece44d55d7b897ad7783d27cac24ed24ae77eeed57b1a3de942d3cb5"} Oct 30 20:59:35.393760 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:35.393706 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ggbwl" event={"ID":"8bc258c0-5d93-4ddb-b49c-9a06167f6185","Type":"ContainerStarted","Data":"e19a827d04e419c8553f530ce0591dd5284d48acb9b1ae403d0ced5d7909c7ba"} Oct 30 20:59:35.395226 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:35.395143 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mgsr5" event={"ID":"5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e","Type":"ContainerStarted","Data":"03dab2bd4f5ccd547d64bb7a5331444dfb48a537d0ebfa55136fc411976c629d"} Oct 30 20:59:35.397844 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:35.397767 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ww5pw" event={"ID":"97872ff8-a1f5-4062-8202-41aa9d8a93bc","Type":"ContainerStarted","Data":"c9e921e24c64a15435106455241e0ff8bfd38a668102ecd4b56609cc4ed29f10"} Oct 30 20:59:35.399395 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:35.399320 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cnj4q" event={"ID":"1362c712-7332-4a6e-a215-633d2f02acca","Type":"ContainerStarted","Data":"bf089dbea2ba22737b1bb71e42206d667d7b745dcaa421f6d6abca101caa3ceb"} Oct 30 20:59:35.404192 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:35.403669 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-3.ec2.internal" event={"ID":"10ed65d1de0a53fd4cc159ce5efb836d","Type":"ContainerStarted","Data":"e6515524026e82ce86fcb69cc026ff764b2ab091e9e8a238f2aaf15ab91f86e2"} Oct 30 20:59:35.406909 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:35.406833 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" event={"ID":"93f2c084-a1d2-4bda-8a41-c01829cc81ba","Type":"ContainerStarted","Data":"c67809ee48958159646ad059a290b0559a8e206498105424fc67d98b5537cd83"} Oct 30 20:59:35.409762 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:35.409740 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gv9s" event={"ID":"03e64eb3-e7df-464d-9382-47cd0e570456","Type":"ContainerStarted","Data":"fcc5c3c5eb56c860c84b8d31c502139f42a747a5ad1b6e0105faf9366a948c44"} Oct 30 20:59:35.413835 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:35.413774 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-54qxj" event={"ID":"a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31","Type":"ContainerStarted","Data":"4961a255689d4f8b3ca3f78ac06b9076bfbd9659ab98826173df297fdb40eda9"} Oct 30 20:59:35.479491 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:35.479419 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-3.ec2.internal" podStartSLOduration=1.479398506 podStartE2EDuration="1.479398506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:35.478459577 +0000 UTC m=+3.595390777" watchObservedRunningTime="2026-10-30 20:59:35.479398506 +0000 UTC m=+3.596329711" Oct 30 20:59:35.975717 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:35.975674 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs\") pod \"network-metrics-daemon-7d6gs\" (UID: \"f6337834-db43-4b8e-a955-9691f72e0f19\") " pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:35.975888 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:35.975779 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:35.975938 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:35.975919 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:59:35.976064 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:35.975981 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret podName:5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e nodeName:}" failed. No retries permitted until 2026-10-30 20:59:37.975963413 +0000 UTC m=+6.092894609 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret") pod "global-pull-secret-syncer-m9dm5" (UID: "5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:59:35.976463 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:35.976439 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:59:35.976551 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:35.976500 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs podName:f6337834-db43-4b8e-a955-9691f72e0f19 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:37.976483608 +0000 UTC m=+6.093414788 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs") pod "network-metrics-daemon-7d6gs" (UID: "f6337834-db43-4b8e-a955-9691f72e0f19") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:59:36.077381 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:36.076729 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz6tt\" (UniqueName: \"kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt\") pod \"network-check-target-mfgjb\" (UID: \"4e46662c-bfe6-45f8-8a23-89ea5fc92ae5\") " pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 20:59:36.077381 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:36.076905 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:59:36.077381 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:36.076929 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:59:36.077381 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:36.076942 2577 projected.go:196] Error preparing data for projected volume kube-api-access-dz6tt for pod openshift-network-diagnostics/network-check-target-mfgjb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:59:36.077381 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:36.077010 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt podName:4e46662c-bfe6-45f8-8a23-89ea5fc92ae5 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:38.0769902 +0000 UTC m=+6.193921396 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dz6tt" (UniqueName: "kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt") pod "network-check-target-mfgjb" (UID: "4e46662c-bfe6-45f8-8a23-89ea5fc92ae5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:59:36.300446 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:36.300301 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:54:34 +0000 UTC" deadline="2028-04-25 15:26:46.965278459 +0000 UTC" Oct 30 20:59:36.300446 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:36.300367 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13026h27m10.664915924s" Oct 30 20:59:36.375879 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:36.375846 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 20:59:36.376039 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:36.375982 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfgjb" podUID="4e46662c-bfe6-45f8-8a23-89ea5fc92ae5" Oct 30 20:59:36.376278 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:36.376261 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:36.376369 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:36.376293 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:36.376428 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:36.376360 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9dm5" podUID="5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e" Oct 30 20:59:36.376487 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:36.376449 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7d6gs" podUID="f6337834-db43-4b8e-a955-9691f72e0f19" Oct 30 20:59:36.429851 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:36.429813 2577 generic.go:356] "Generic (PLEG): container finished" podID="86fd282564aec729bf61e75aa114744a" containerID="3ed8e2409579bd986770f5c33540d4899bae49deadbd613baab5f08db05729cd" exitCode=0 Oct 30 20:59:36.430172 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:36.430149 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" event={"ID":"86fd282564aec729bf61e75aa114744a","Type":"ContainerDied","Data":"3ed8e2409579bd986770f5c33540d4899bae49deadbd613baab5f08db05729cd"} Oct 30 20:59:37.455971 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:37.455225 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" event={"ID":"86fd282564aec729bf61e75aa114744a","Type":"ContainerStarted","Data":"1beab69337c9ae8002f086417000c8864d06ef0bcd0c69d8141f51b0cdef4f5f"} Oct 30 20:59:37.995249 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:37.994378 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:37.995249 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:37.994439 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs\") pod \"network-metrics-daemon-7d6gs\" (UID: \"f6337834-db43-4b8e-a955-9691f72e0f19\") " pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:37.995249 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:37.994601 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:59:37.995249 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:37.994667 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs podName:f6337834-db43-4b8e-a955-9691f72e0f19 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:41.994646052 +0000 UTC m=+10.111577246 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs") pod "network-metrics-daemon-7d6gs" (UID: "f6337834-db43-4b8e-a955-9691f72e0f19") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:59:37.995249 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:37.995071 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:59:37.995249 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:37.995143 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret podName:5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e nodeName:}" failed. No retries permitted until 2026-10-30 20:59:41.995126413 +0000 UTC m=+10.112057612 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret") pod "global-pull-secret-syncer-m9dm5" (UID: "5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:59:38.095613 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:38.094912 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz6tt\" (UniqueName: \"kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt\") pod \"network-check-target-mfgjb\" (UID: \"4e46662c-bfe6-45f8-8a23-89ea5fc92ae5\") " pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 20:59:38.095613 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:38.095148 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:59:38.095613 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:38.095169 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:59:38.095613 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:38.095183 2577 projected.go:196] Error preparing data for projected volume kube-api-access-dz6tt for pod openshift-network-diagnostics/network-check-target-mfgjb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:59:38.095613 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:38.095242 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt podName:4e46662c-bfe6-45f8-8a23-89ea5fc92ae5 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:42.095223208 +0000 UTC m=+10.212154435 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dz6tt" (UniqueName: "kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt") pod "network-check-target-mfgjb" (UID: "4e46662c-bfe6-45f8-8a23-89ea5fc92ae5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:59:38.369037 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:38.368267 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:38.369037 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:38.368298 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 20:59:38.369037 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:38.368396 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9dm5" podUID="5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e" Oct 30 20:59:38.369037 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:38.368816 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfgjb" podUID="4e46662c-bfe6-45f8-8a23-89ea5fc92ae5" Oct 30 20:59:38.369037 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:38.368868 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:38.369037 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:38.368952 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7d6gs" podUID="f6337834-db43-4b8e-a955-9691f72e0f19" Oct 30 20:59:38.464240 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:38.462795 2577 generic.go:356] "Generic (PLEG): container finished" podID="03e64eb3-e7df-464d-9382-47cd0e570456" containerID="7d4d7e8045f383fb54cd7a6b7dce08070023d6b4df2a9698c17f158f5ad9407e" exitCode=0 Oct 30 20:59:38.464240 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:38.463912 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gv9s" event={"ID":"03e64eb3-e7df-464d-9382-47cd0e570456","Type":"ContainerDied","Data":"7d4d7e8045f383fb54cd7a6b7dce08070023d6b4df2a9698c17f158f5ad9407e"} Oct 30 20:59:38.535126 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:38.534435 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" podStartSLOduration=4.534418583 podStartE2EDuration="4.534418583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:59:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:37.495215758 +0000 UTC m=+5.612146962" watchObservedRunningTime="2026-10-30 20:59:38.534418583 +0000 UTC m=+6.651349785" Oct 30 20:59:39.473421 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:39.472479 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mgsr5" event={"ID":"5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e","Type":"ContainerStarted","Data":"1898568914d5e5d6275c850d900113c4d68484c262a426811e855cffe865b269"} Oct 30 20:59:39.481210 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:39.480695 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cnj4q" event={"ID":"1362c712-7332-4a6e-a215-633d2f02acca","Type":"ContainerStarted","Data":"aa35aaca3bbff1b3b77a4d3e906458c57df68a620da9dc90c4b1a88baeb30c08"} Oct 30 20:59:39.550974 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:39.550181 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mgsr5" podStartSLOduration=2.746430246 podStartE2EDuration="6.550159778s" totalImagesPullingTime="3.803729532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:39.511880602 +0000 UTC m=+7.628811799" watchObservedRunningTime="2026-10-30 20:59:39.550159778 +0000 UTC m=+7.667090981" Oct 30 20:59:40.368772 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:40.368741 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 20:59:40.368903 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:40.368874 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfgjb" podUID="4e46662c-bfe6-45f8-8a23-89ea5fc92ae5" Oct 30 20:59:40.368982 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:40.368968 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:40.369392 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:40.369127 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9dm5" podUID="5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e" Oct 30 20:59:40.369392 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:40.369200 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:40.369392 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:40.369290 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7d6gs" podUID="f6337834-db43-4b8e-a955-9691f72e0f19" Oct 30 20:59:40.486659 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:40.486577 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" event={"ID":"93f2c084-a1d2-4bda-8a41-c01829cc81ba","Type":"ContainerStarted","Data":"e1c8a42e6ca0588b50ef382bc4db9d9160aec7e22d5a4f81e4ecdad04432ce3a"} Oct 30 20:59:41.930067 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:41.929995 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 20:59:42.029466 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:42.029387 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:42.029466 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:42.029448 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs\") pod \"network-metrics-daemon-7d6gs\" (UID: \"f6337834-db43-4b8e-a955-9691f72e0f19\") " pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:42.029626 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:42.029576 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:59:42.029655 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:42.029634 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs podName:f6337834-db43-4b8e-a955-9691f72e0f19 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:50.029616461 +0000 UTC m=+18.146547640 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs") pod "network-metrics-daemon-7d6gs" (UID: "f6337834-db43-4b8e-a955-9691f72e0f19") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:59:42.029694 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:42.029688 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:59:42.029724 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:42.029714 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret podName:5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e nodeName:}" failed. No retries permitted until 2026-10-30 20:59:50.029706317 +0000 UTC m=+18.146637508 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret") pod "global-pull-secret-syncer-m9dm5" (UID: "5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:59:42.130982 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:42.130324 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz6tt\" (UniqueName: \"kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt\") pod \"network-check-target-mfgjb\" (UID: \"4e46662c-bfe6-45f8-8a23-89ea5fc92ae5\") " pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 20:59:42.130982 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:42.130537 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:59:42.130982 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:42.130562 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:59:42.130982 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:42.130575 2577 projected.go:196] Error preparing data for projected volume kube-api-access-dz6tt for pod openshift-network-diagnostics/network-check-target-mfgjb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:59:42.130982 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:42.130632 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt podName:4e46662c-bfe6-45f8-8a23-89ea5fc92ae5 nodeName:}" failed. No retries permitted until 2026-10-30 20:59:50.130613896 +0000 UTC m=+18.247545088 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dz6tt" (UniqueName: "kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt") pod "network-check-target-mfgjb" (UID: "4e46662c-bfe6-45f8-8a23-89ea5fc92ae5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:59:42.310557 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:42.309726 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T20:59:41.930014977Z","UUID":"f7e1a4c9-eab7-4af7-85a4-b8f9c6ae1c3e","Handler":null,"Name":"","Endpoint":""} Oct 30 20:59:42.317111 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:42.316267 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 20:59:42.317111 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:42.316296 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 20:59:42.367594 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:42.367565 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 20:59:42.367720 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:42.367648 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfgjb" podUID="4e46662c-bfe6-45f8-8a23-89ea5fc92ae5" Oct 30 20:59:42.367769 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:42.367736 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:42.367801 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:42.367769 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:42.367899 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:42.367879 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7d6gs" podUID="f6337834-db43-4b8e-a955-9691f72e0f19" Oct 30 20:59:42.367964 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:42.367948 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9dm5" podUID="5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e" Oct 30 20:59:42.496595 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:42.496566 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" event={"ID":"93f2c084-a1d2-4bda-8a41-c01829cc81ba","Type":"ContainerStarted","Data":"36521789203e5bf4dca900598e8ad1934e7bec63a00632d20350fc1e1e8fdc0b"} Oct 30 20:59:43.337840 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:43.337803 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mgsr5" Oct 30 20:59:43.338626 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:43.338609 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mgsr5" Oct 30 20:59:43.373636 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:43.373306 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cnj4q" podStartSLOduration=5.801646338 podStartE2EDuration="10.37328992s" totalImagesPullingTime="4.571643582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:39.550880261 +0000 UTC m=+7.667811463" watchObservedRunningTime="2026-10-30 20:59:43.37328992 +0000 UTC m=+11.490221119" Oct 30 20:59:43.499815 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:43.499779 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ggbwl" event={"ID":"8bc258c0-5d93-4ddb-b49c-9a06167f6185","Type":"ContainerStarted","Data":"704ca5286e5eb75da2ff6c1d1b58f525d91d8b16d4b79cf1fb8b462ecf791d18"} Oct 30 20:59:43.503024 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:43.502993 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" event={"ID":"93f2c084-a1d2-4bda-8a41-c01829cc81ba","Type":"ContainerStarted","Data":"7ce82bac30bf9c1532b7554efa96fb348f0b7614cee3446589732c16b7172c14"} Oct 30 20:59:43.505494 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:43.505468 2577 generic.go:356] "Generic (PLEG): container finished" podID="03e64eb3-e7df-464d-9382-47cd0e570456" containerID="8c66728984e57caf7e21a9a9cf706207b308ea9a73b6db26767afd57f87d5ce7" exitCode=0 Oct 30 20:59:43.505711 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:43.505681 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gv9s" event={"ID":"03e64eb3-e7df-464d-9382-47cd0e570456","Type":"ContainerDied","Data":"8c66728984e57caf7e21a9a9cf706207b308ea9a73b6db26767afd57f87d5ce7"} Oct 30 20:59:43.507490 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:43.507193 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-54qxj" event={"ID":"a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31","Type":"ContainerStarted","Data":"cab51f48e81ec50947d686e34ceb4630f260eb5d9b08544279cd3eef4e273f7f"} Oct 30 20:59:43.537322 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:43.537277 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-ggbwl" podStartSLOduration=2.699301787 podStartE2EDuration="10.537258984s" totalImagesPullingTime="7.837957197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:43.536763577 +0000 UTC m=+11.653694779" watchObservedRunningTime="2026-10-30 20:59:43.537258984 +0000 UTC m=+11.654190184" Oct 30 20:59:43.573719 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:43.573538 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" podStartSLOduration=2.558219843 podStartE2EDuration="10.573523716s" totalImagesPullingTime="8.015303873s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:59:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:43.573114934 +0000 UTC m=+11.690046133" watchObservedRunningTime="2026-10-30 20:59:43.573523716 +0000 UTC m=+11.690454922" Oct 30 20:59:43.611120 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:43.611080 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-54qxj" podStartSLOduration=2.69323349 podStartE2EDuration="10.6110663s" totalImagesPullingTime="7.91783281s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:43.610632193 +0000 UTC m=+11.727563393" watchObservedRunningTime="2026-10-30 20:59:43.6110663 +0000 UTC m=+11.727997498" Oct 30 20:59:44.369635 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:44.369607 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:44.369973 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:44.369739 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9dm5" podUID="5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e" Oct 30 20:59:44.370153 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:44.370136 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:44.370269 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:44.370250 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7d6gs" podUID="f6337834-db43-4b8e-a955-9691f72e0f19" Oct 30 20:59:44.370328 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:44.370315 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 20:59:44.370403 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:44.370382 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfgjb" podUID="4e46662c-bfe6-45f8-8a23-89ea5fc92ae5" Oct 30 20:59:44.512344 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:44.512314 2577 generic.go:356] "Generic (PLEG): container finished" podID="03e64eb3-e7df-464d-9382-47cd0e570456" containerID="4c2220296aeccf5ffcd6081399bb63ad9567023d8ed635ad356eaffb71383ff1" exitCode=0 Oct 30 20:59:44.512471 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:44.512399 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gv9s" event={"ID":"03e64eb3-e7df-464d-9382-47cd0e570456","Type":"ContainerDied","Data":"4c2220296aeccf5ffcd6081399bb63ad9567023d8ed635ad356eaffb71383ff1"} Oct 30 20:59:44.514047 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:44.514011 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ww5pw" event={"ID":"97872ff8-a1f5-4062-8202-41aa9d8a93bc","Type":"ContainerStarted","Data":"f70fd5b043803535c97f8c80c8ad27c562f76aac1a908d3802f67143ea5e08b6"} Oct 30 20:59:44.590213 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:44.590162 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ww5pw" podStartSLOduration=3.383353438 podStartE2EDuration="11.590145829s" totalImagesPullingTime="8.206792391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:44.590021885 +0000 UTC m=+12.706953084" watchObservedRunningTime="2026-10-30 20:59:44.590145829 +0000 UTC m=+12.707077029" Oct 30 20:59:44.835427 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:44.835406 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mgsr5" Oct 30 20:59:44.835551 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:44.835517 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 20:59:44.836155 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:44.835948 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mgsr5" Oct 30 20:59:45.519049 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:45.518952 2577 generic.go:356] "Generic (PLEG): container finished" podID="03e64eb3-e7df-464d-9382-47cd0e570456" containerID="2e9016d861ab1a8ba80f2f74331308204bb48fe7ff770da101c1570bee7eb698" exitCode=0 Oct 30 20:59:45.519676 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:45.519057 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gv9s" event={"ID":"03e64eb3-e7df-464d-9382-47cd0e570456","Type":"ContainerDied","Data":"2e9016d861ab1a8ba80f2f74331308204bb48fe7ff770da101c1570bee7eb698"} Oct 30 20:59:45.520713 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:45.520685 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" event={"ID":"1e20348d-9970-4de7-80be-84f29ef8ece6","Type":"ContainerStarted","Data":"891bd14c1c9b0700a682654c3aa2b6af819abed53efac87fc7059b29e677900f"} Oct 30 20:59:45.595867 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:45.595805 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" podStartSLOduration=2.428116112 podStartE2EDuration="12.59578757s" totalImagesPullingTime="10.167671458s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:45.595678123 +0000 UTC m=+13.712609322" watchObservedRunningTime="2026-10-30 20:59:45.59578757 +0000 UTC m=+13.712718769" Oct 30 20:59:46.367180 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:46.367146 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:46.367357 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:46.367214 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:46.367357 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:46.367311 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 20:59:46.367357 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:46.367323 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9dm5" podUID="5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e" Oct 30 20:59:46.367510 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:46.367384 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfgjb" podUID="4e46662c-bfe6-45f8-8a23-89ea5fc92ae5" Oct 30 20:59:46.367510 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:46.367457 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7d6gs" podUID="f6337834-db43-4b8e-a955-9691f72e0f19" Oct 30 20:59:47.529235 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:47.529203 2577 generic.go:356] "Generic (PLEG): container finished" podID="03e64eb3-e7df-464d-9382-47cd0e570456" containerID="840016763d62b059579856ba431dbd0df93a21a1c5c14715b6d7b015ae36383d" exitCode=0 Oct 30 20:59:47.529828 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:47.529248 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gv9s" event={"ID":"03e64eb3-e7df-464d-9382-47cd0e570456","Type":"ContainerDied","Data":"840016763d62b059579856ba431dbd0df93a21a1c5c14715b6d7b015ae36383d"} Oct 30 20:59:48.367217 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:48.367180 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 20:59:48.367378 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:48.367298 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfgjb" podUID="4e46662c-bfe6-45f8-8a23-89ea5fc92ae5" Oct 30 20:59:48.367660 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:48.367641 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:48.367742 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:48.367728 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9dm5" podUID="5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e" Oct 30 20:59:48.367779 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:48.367757 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:48.367820 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:48.367809 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7d6gs" podUID="f6337834-db43-4b8e-a955-9691f72e0f19" Oct 30 20:59:48.533770 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:48.533736 2577 generic.go:356] "Generic (PLEG): container finished" podID="03e64eb3-e7df-464d-9382-47cd0e570456" containerID="909bdfaa047b2ffe9cf2079a75ea39471b3de7ea9bf8fd5c7653f8c06ec5cec1" exitCode=0 Oct 30 20:59:48.534153 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:48.533777 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gv9s" event={"ID":"03e64eb3-e7df-464d-9382-47cd0e570456","Type":"ContainerDied","Data":"909bdfaa047b2ffe9cf2079a75ea39471b3de7ea9bf8fd5c7653f8c06ec5cec1"} Oct 30 20:59:49.539112 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:49.538943 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9gv9s" event={"ID":"03e64eb3-e7df-464d-9382-47cd0e570456","Type":"ContainerStarted","Data":"fc64093c6cd125c02063e3717004766f7acd75b866d60b1c5cefa2f9be4fd4c9"} Oct 30 20:59:49.543702 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:49.543644 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" event={"ID":"88f212cc-4eb8-4a49-9c13-a1d395ab43c6","Type":"ContainerStarted","Data":"0fe3a39bfbda66d6a06bfa5286101b825a04e5f08867d954f92fcbf792d1d3cb"} Oct 30 20:59:49.543797 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:49.543717 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" event={"ID":"88f212cc-4eb8-4a49-9c13-a1d395ab43c6","Type":"ContainerStarted","Data":"57e1d197b7f1ca8b83e130bdcedff6cb6d5744645ca0f7cc216ad54d876b61a7"} Oct 30 20:59:49.543797 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:49.543770 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" event={"ID":"88f212cc-4eb8-4a49-9c13-a1d395ab43c6","Type":"ContainerStarted","Data":"9db457f582f3363bb78c6ac916695481a6be1738c7cd2fd47dc0ca982830ddf6"} Oct 30 20:59:49.543894 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:49.543815 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" event={"ID":"88f212cc-4eb8-4a49-9c13-a1d395ab43c6","Type":"ContainerStarted","Data":"f3ea1f79d547e5ea242cb36647fefd7d00e59485e570ac59e5ac4aa39789118b"} Oct 30 20:59:49.597899 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:49.597853 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9gv9s" podStartSLOduration=6.510978982 podStartE2EDuration="16.597836944s" totalImagesPullingTime="10.086857962s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:59:33 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:49.590790822 +0000 UTC m=+17.707722021" watchObservedRunningTime="2026-10-30 20:59:49.597836944 +0000 UTC m=+17.714768145" Oct 30 20:59:50.100910 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:50.100887 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:50.101108 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:50.100930 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs\") pod \"network-metrics-daemon-7d6gs\" (UID: \"f6337834-db43-4b8e-a955-9691f72e0f19\") " pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:50.101108 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:50.101058 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:59:50.101108 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:50.101105 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs podName:f6337834-db43-4b8e-a955-9691f72e0f19 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:06.101093375 +0000 UTC m=+34.218024552 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs") pod "network-metrics-daemon-7d6gs" (UID: "f6337834-db43-4b8e-a955-9691f72e0f19") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 20:59:50.101257 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:50.101058 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 20:59:50.101257 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:50.101186 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret podName:5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e nodeName:}" failed. No retries permitted until 2026-10-30 21:00:06.101169715 +0000 UTC m=+34.218100895 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret") pod "global-pull-secret-syncer-m9dm5" (UID: "5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e") : object "kube-system"/"original-pull-secret" not registered Oct 30 20:59:50.201397 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:50.201364 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz6tt\" (UniqueName: \"kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt\") pod \"network-check-target-mfgjb\" (UID: \"4e46662c-bfe6-45f8-8a23-89ea5fc92ae5\") " pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 20:59:50.201537 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:50.201516 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 20:59:50.201577 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:50.201541 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 20:59:50.201577 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:50.201551 2577 projected.go:196] Error preparing data for projected volume kube-api-access-dz6tt for pod openshift-network-diagnostics/network-check-target-mfgjb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:59:50.201647 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:50.201598 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt podName:4e46662c-bfe6-45f8-8a23-89ea5fc92ae5 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:06.201582434 +0000 UTC m=+34.318513615 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dz6tt" (UniqueName: "kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt") pod "network-check-target-mfgjb" (UID: "4e46662c-bfe6-45f8-8a23-89ea5fc92ae5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 20:59:50.367244 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:50.367178 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:50.367374 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:50.367280 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:50.367374 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:50.367296 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9dm5" podUID="5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e" Oct 30 20:59:50.367374 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:50.367366 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7d6gs" podUID="f6337834-db43-4b8e-a955-9691f72e0f19" Oct 30 20:59:50.367489 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:50.367382 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 20:59:50.367522 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:50.367481 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfgjb" podUID="4e46662c-bfe6-45f8-8a23-89ea5fc92ae5" Oct 30 20:59:50.550047 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:50.550000 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" event={"ID":"88f212cc-4eb8-4a49-9c13-a1d395ab43c6","Type":"ContainerStarted","Data":"40a972b9a195f617372da825bb4e3eec4ab9274851660a19ae4ed6e48113b4d7"} Oct 30 20:59:50.550420 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:50.550102 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" event={"ID":"88f212cc-4eb8-4a49-9c13-a1d395ab43c6","Type":"ContainerStarted","Data":"1dd53c73cf767f6776a1e0537399f7faefd5c16fc3e2c3e5bc840f39ff1cfe3c"} Oct 30 20:59:52.367396 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:52.367370 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:52.367852 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:52.367455 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9dm5" podUID="5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e" Oct 30 20:59:52.367852 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:52.367545 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 20:59:52.367852 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:52.367627 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfgjb" podUID="4e46662c-bfe6-45f8-8a23-89ea5fc92ae5" Oct 30 20:59:52.367852 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:52.367675 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:52.367852 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:52.367742 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7d6gs" podUID="f6337834-db43-4b8e-a955-9691f72e0f19" Oct 30 20:59:52.559561 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:52.559530 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" event={"ID":"88f212cc-4eb8-4a49-9c13-a1d395ab43c6","Type":"ContainerStarted","Data":"239ea58a21882fefb470c6d41d7606f71e73ea9881dcdab11b7c860f56c2b789"} Oct 30 20:59:54.366907 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:54.366874 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:54.367536 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:54.366945 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:54.367536 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:54.366945 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 20:59:54.367536 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:54.367080 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7d6gs" podUID="f6337834-db43-4b8e-a955-9691f72e0f19" Oct 30 20:59:54.367536 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:54.367102 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9dm5" podUID="5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e" Oct 30 20:59:54.367536 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:54.367148 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfgjb" podUID="4e46662c-bfe6-45f8-8a23-89ea5fc92ae5" Oct 30 20:59:54.565943 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:54.565898 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" event={"ID":"88f212cc-4eb8-4a49-9c13-a1d395ab43c6","Type":"ContainerStarted","Data":"19f685882cfef5a6a747569c1cc5d3b655bab576bff45eba5989afcdcc6727c5"} Oct 30 20:59:54.566176 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:54.566160 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:54.580975 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:54.580954 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:54.642780 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:54.642709 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" podStartSLOduration=7.004453682 podStartE2EDuration="21.642694978s" totalImagesPullingTime="14.638241296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:59:54.642347838 +0000 UTC m=+22.759279036" watchObservedRunningTime="2026-10-30 20:59:54.642694978 +0000 UTC m=+22.759626177" Oct 30 20:59:55.568428 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:55.568250 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:55.568856 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:55.568472 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:55.582598 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:55.582579 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 20:59:55.710317 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:55.710288 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m9dm5"] Oct 30 20:59:55.710478 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:55.710435 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:55.710562 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:55.710539 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9dm5" podUID="5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e" Oct 30 20:59:55.713809 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:55.713783 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mfgjb"] Oct 30 20:59:55.713947 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:55.713893 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 20:59:55.714012 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:55.713992 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfgjb" podUID="4e46662c-bfe6-45f8-8a23-89ea5fc92ae5" Oct 30 20:59:55.714414 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:55.714396 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7d6gs"] Oct 30 20:59:55.714504 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:55.714495 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:55.714606 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:55.714588 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7d6gs" podUID="f6337834-db43-4b8e-a955-9691f72e0f19" Oct 30 20:59:57.366551 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:57.366521 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 20:59:57.366986 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:57.366560 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:57.366986 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:57.366667 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfgjb" podUID="4e46662c-bfe6-45f8-8a23-89ea5fc92ae5" Oct 30 20:59:57.366986 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:57.366699 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9dm5" podUID="5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e" Oct 30 20:59:57.366986 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:57.366735 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:57.366986 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:57.366808 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7d6gs" podUID="f6337834-db43-4b8e-a955-9691f72e0f19" Oct 30 20:59:59.366486 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:59.366455 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 20:59:59.366486 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:59.366477 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 20:59:59.366943 ip-10-0-142-3 kubenswrapper[2577]: I1030 20:59:59.366458 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 20:59:59.366943 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:59.366542 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfgjb" podUID="4e46662c-bfe6-45f8-8a23-89ea5fc92ae5" Oct 30 20:59:59.366943 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:59.366623 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9dm5" podUID="5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e" Oct 30 20:59:59.366943 ip-10-0-142-3 kubenswrapper[2577]: E1030 20:59:59.366690 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7d6gs" podUID="f6337834-db43-4b8e-a955-9691f72e0f19" Oct 30 21:00:01.366631 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:01.366600 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 21:00:01.366631 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:01.366631 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 21:00:01.367130 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:01.366707 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 21:00:01.367130 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:01.366717 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7d6gs" podUID="f6337834-db43-4b8e-a955-9691f72e0f19" Oct 30 21:00:01.367130 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:01.366761 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mfgjb" podUID="4e46662c-bfe6-45f8-8a23-89ea5fc92ae5" Oct 30 21:00:01.367130 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:01.366814 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m9dm5" podUID="5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e" Oct 30 21:00:02.172314 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.172288 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeReady" Oct 30 21:00:02.172459 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.172386 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:00:02.215338 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.215311 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zdvb2"] Oct 30 21:00:02.217127 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.217112 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zdvb2" Oct 30 21:00:02.218885 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.218860 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zr75n"] Oct 30 21:00:02.220760 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.220714 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-b9fhn"] Oct 30 21:00:02.220876 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.220821 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zr75n" Oct 30 21:00:02.222185 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.222169 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87c42"] Oct 30 21:00:02.222273 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.222261 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-b9fhn" Oct 30 21:00:02.223576 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.223561 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm"] Oct 30 21:00:02.223694 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.223658 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87c42" Oct 30 21:00:02.225123 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.225105 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-m7m4c"] Oct 30 21:00:02.225212 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.225161 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm" Oct 30 21:00:02.226570 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.226552 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx"] Oct 30 21:00:02.226660 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.226637 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:02.228258 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.228238 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" Oct 30 21:00:02.234436 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.233074 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56886949f8-xwhft"] Oct 30 21:00:02.235606 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.235588 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-c5khs"] Oct 30 21:00:02.235694 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.235671 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.238895 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.238878 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz"] Oct 30 21:00:02.238999 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.238976 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:02.240840 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.240724 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-758fcd4795-j6bts"] Oct 30 21:00:02.240840 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.240799 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:02.242745 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.242726 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6789dbc4b5-5gqgl"] Oct 30 21:00:02.242834 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.242780 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.243766 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.243746 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 21:00:02.247545 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.244376 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 21:00:02.247545 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.244776 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7"] Oct 30 21:00:02.247545 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.245704 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:02.248419 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.248285 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-j5tcr"] Oct 30 21:00:02.248419 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.248315 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:00:02.248419 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.248389 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 21:00:02.248657 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.248495 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-kmr6z\"" Oct 30 21:00:02.249171 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.249149 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7" Oct 30 21:00:02.251144 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.251125 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-wbrhm"] Oct 30 21:00:02.251250 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.251235 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j5tcr" Oct 30 21:00:02.253804 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.253783 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bkjxf\"" Oct 30 21:00:02.255435 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.255415 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zdvb2"] Oct 30 21:00:02.256684 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.256381 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 21:00:02.257447 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.257131 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mmzzs"] Oct 30 21:00:02.257594 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.257574 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" Oct 30 21:00:02.263269 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.262298 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-888xx"] Oct 30 21:00:02.263392 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.263374 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mmzzs" Oct 30 21:00:02.265978 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.265960 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-b9fhn"] Oct 30 21:00:02.266118 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.266098 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-888xx" Oct 30 21:00:02.266430 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.266123 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zr75n"] Oct 30 21:00:02.266430 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.266149 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-m7m4c"] Oct 30 21:00:02.266430 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.266169 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87c42"] Oct 30 21:00:02.266430 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.266188 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx"] Oct 30 21:00:02.266430 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.266211 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz"] Oct 30 21:00:02.266430 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.266229 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-c5khs"] Oct 30 21:00:02.266430 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.266248 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-758fcd4795-j6bts"] Oct 30 21:00:02.266430 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.266273 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-j5tcr"] Oct 30 21:00:02.274458 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.274438 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-888xx"] Oct 30 21:00:02.277000 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.276971 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm"] Oct 30 21:00:02.281292 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.281275 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7"] Oct 30 21:00:02.286025 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.286003 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56886949f8-xwhft"] Oct 30 21:00:02.291464 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.288810 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-wbrhm"] Oct 30 21:00:02.293863 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.293754 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2mms\" (UniqueName: \"kubernetes.io/projected/a66e14f7-3c72-48a5-88e8-e9f78a7c7ff5-kube-api-access-h2mms\") pod \"volume-data-source-validator-58c9f5f46d-87c42\" (UID: \"a66e14f7-3c72-48a5-88e8-e9f78a7c7ff5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87c42" Oct 30 21:00:02.293967 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.293908 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/705e2b0e-10e1-4710-8dfb-bbd862c8c084-serving-cert\") pod \"service-ca-operator-544d749c8c-vprbm\" (UID: \"705e2b0e-10e1-4710-8dfb-bbd862c8c084\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm" Oct 30 21:00:02.293967 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.293947 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-snapshots\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:02.294093 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.293989 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2k65\" (UniqueName: \"kubernetes.io/projected/0f0b2878-417e-4986-8067-b50a16d16967-kube-api-access-z2k65\") pod \"kas-connection-checker-685895c7c9-zdvb2\" (UID: \"0f0b2878-417e-4986-8067-b50a16d16967\") " pod="kube-system/kas-connection-checker-685895c7c9-zdvb2" Oct 30 21:00:02.294093 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.294069 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxsnl\" (UniqueName: \"kubernetes.io/projected/c0b2deee-2b0c-4c2c-9ac5-debd33f12ba3-kube-api-access-nxsnl\") pod \"kas-connection-checker-685895c7c9-b9fhn\" (UID: \"c0b2deee-2b0c-4c2c-9ac5-debd33f12ba3\") " pod="kube-system/kas-connection-checker-685895c7c9-b9fhn" Oct 30 21:00:02.294192 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.294118 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b67sr\" (UniqueName: \"kubernetes.io/projected/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-kube-api-access-b67sr\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:02.294192 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.294183 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/705e2b0e-10e1-4710-8dfb-bbd862c8c084-config\") pod \"service-ca-operator-544d749c8c-vprbm\" (UID: \"705e2b0e-10e1-4710-8dfb-bbd862c8c084\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm" Oct 30 21:00:02.294334 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.294230 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6gqs\" (UniqueName: \"kubernetes.io/projected/705e2b0e-10e1-4710-8dfb-bbd862c8c084-kube-api-access-x6gqs\") pod \"service-ca-operator-544d749c8c-vprbm\" (UID: \"705e2b0e-10e1-4710-8dfb-bbd862c8c084\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm" Oct 30 21:00:02.294399 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.294359 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:02.294494 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.294463 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-serving-cert\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:02.294587 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.294554 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmcw2\" (UniqueName: \"kubernetes.io/projected/6d9c7935-1d3d-4323-ac72-008a394e7f31-kube-api-access-nmcw2\") pod \"kas-connection-checker-685895c7c9-zr75n\" (UID: \"6d9c7935-1d3d-4323-ac72-008a394e7f31\") " pod="kube-system/kas-connection-checker-685895c7c9-zr75n" Oct 30 21:00:02.294671 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.294648 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-service-ca-bundle\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:02.294728 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.294712 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-tmp\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:02.294782 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.294761 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5r7xx\" (UID: \"ef2c01c8-12ef-4294-963e-4e840e2eecff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" Oct 30 21:00:02.294951 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.294932 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5r7xx\" (UID: \"ef2c01c8-12ef-4294-963e-4e840e2eecff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" Oct 30 21:00:02.295045 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.294990 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hfkg\" (UniqueName: \"kubernetes.io/projected/ef2c01c8-12ef-4294-963e-4e840e2eecff-kube-api-access-9hfkg\") pod \"cluster-samples-operator-6ddb6c8b94-5r7xx\" (UID: \"ef2c01c8-12ef-4294-963e-4e840e2eecff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" Oct 30 21:00:02.296226 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.296204 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mmzzs"] Oct 30 21:00:02.296691 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.296674 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6789dbc4b5-5gqgl"] Oct 30 21:00:02.343425 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.343396 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:00:02.343512 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.343458 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:00:02.343512 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.343505 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 21:00:02.343585 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.343556 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 21:00:02.343585 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.343569 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 21:00:02.343585 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.343569 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 21:00:02.343713 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.343592 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:00:02.343713 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.343598 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:00:02.343835 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.343819 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 21:00:02.343898 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.343846 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:00:02.343977 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.343954 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:00:02.344045 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.343985 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 21:00:02.344045 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.344025 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:00:02.344142 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.344054 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:00:02.344142 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.344068 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:00:02.344230 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.344186 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:00:02.344230 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.344198 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:00:02.344304 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.344294 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 21:00:02.357613 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.357532 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 21:00:02.358584 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.358567 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:00:02.359143 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.359129 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:00:02.367897 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.367788 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 21:00:02.368477 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.368077 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:00:02.368477 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.368106 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 21:00:02.368477 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.368162 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 21:00:02.368477 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.368174 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 21:00:02.369279 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.369260 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xstbj\"" Oct 30 21:00:02.369483 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.369449 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:00:02.369612 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.369600 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:00:02.370118 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.369784 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:00:02.370118 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.369868 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 21:00:02.370118 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.369902 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 21:00:02.370118 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.369960 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 21:00:02.370118 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.370001 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 21:00:02.370118 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.370083 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:00:02.370118 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.370124 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 21:00:02.371072 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.370178 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 21:00:02.371072 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.370228 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:00:02.371072 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.370322 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-jn4qp\"" Oct 30 21:00:02.371072 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.370379 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:00:02.371072 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.370408 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:00:02.371072 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.370459 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:00:02.371072 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.370497 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 21:00:02.371072 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.370574 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4cglt\"" Oct 30 21:00:02.371643 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.371623 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5jdx7\"" Oct 30 21:00:02.371735 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.371688 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-q5gbc\"" Oct 30 21:00:02.371808 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.371793 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-nnb2b\"" Oct 30 21:00:02.371858 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.371837 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-bxqk5\"" Oct 30 21:00:02.371988 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.371976 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-fnzdx\"" Oct 30 21:00:02.372027 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.371980 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 21:00:02.372127 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.372112 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-blkcc\"" Oct 30 21:00:02.372438 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.372170 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-p9dq9\"" Oct 30 21:00:02.372525 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.372514 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:00:02.372600 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.372586 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-4k2gt\"" Oct 30 21:00:02.372854 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.372833 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 21:00:02.396453 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.396432 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wbrhm\" (UID: \"9ffe8e19-6888-4eca-ae9d-9d0fc05ba752\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" Oct 30 21:00:02.396534 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.396458 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz752\" (UniqueName: \"kubernetes.io/projected/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-kube-api-access-cz752\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:02.396534 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.396475 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b61b643-c3cf-453c-875e-25576445ad4f-ca-trust-extracted\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.396534 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.396489 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-bound-sa-token\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.396534 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.396506 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-tmp\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:02.396680 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.396648 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5r7xx\" (UID: \"ef2c01c8-12ef-4294-963e-4e840e2eecff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" Oct 30 21:00:02.396718 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.396697 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hfkg\" (UniqueName: \"kubernetes.io/projected/ef2c01c8-12ef-4294-963e-4e840e2eecff-kube-api-access-9hfkg\") pod \"cluster-samples-operator-6ddb6c8b94-5r7xx\" (UID: \"ef2c01c8-12ef-4294-963e-4e840e2eecff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" Oct 30 21:00:02.396750 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:02.396740 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:00:02.396788 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.396765 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/662ead00-b524-4a6e-815b-c3399b3c9e6c-tmp-dir\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:02.396788 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.396781 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-serving-cert\") pod \"console-operator-7cd56b8dbc-c5khs\" (UID: \"1a2fb9ac-a905-4e20-b1be-7928d5e7881a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:02.396899 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.396803 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-tmp\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:02.396953 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:02.396905 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-tls podName:ef2c01c8-12ef-4294-963e-4e840e2eecff nodeName:}" failed. No retries permitted until 2026-10-30 21:00:02.896882561 +0000 UTC m=+31.013813766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5r7xx" (UID: "ef2c01c8-12ef-4294-963e-4e840e2eecff") : secret "samples-operator-tls" not found Oct 30 21:00:02.396953 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.396935 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b4ba52c-ad04-4a25-9158-89941367abd3-image-registry-private-configuration\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.397078 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.396972 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2mms\" (UniqueName: \"kubernetes.io/projected/a66e14f7-3c72-48a5-88e8-e9f78a7c7ff5-kube-api-access-h2mms\") pod \"volume-data-source-validator-58c9f5f46d-87c42\" (UID: \"a66e14f7-3c72-48a5-88e8-e9f78a7c7ff5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87c42" Oct 30 21:00:02.397078 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.397005 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b4ba52c-ad04-4a25-9158-89941367abd3-installation-pull-secrets\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.397078 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.397064 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/705e2b0e-10e1-4710-8dfb-bbd862c8c084-serving-cert\") pod \"service-ca-operator-544d749c8c-vprbm\" (UID: \"705e2b0e-10e1-4710-8dfb-bbd862c8c084\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm" Oct 30 21:00:02.397201 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.397090 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/662ead00-b524-4a6e-815b-c3399b3c9e6c-config-volume\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:02.397201 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.397184 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b4ba52c-ad04-4a25-9158-89941367abd3-trusted-ca\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.397267 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.397207 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-trusted-ca\") pod \"console-operator-7cd56b8dbc-c5khs\" (UID: \"1a2fb9ac-a905-4e20-b1be-7928d5e7881a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:02.397267 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.397241 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2k65\" (UniqueName: \"kubernetes.io/projected/0f0b2878-417e-4986-8067-b50a16d16967-kube-api-access-z2k65\") pod \"kas-connection-checker-685895c7c9-zdvb2\" (UID: \"0f0b2878-417e-4986-8067-b50a16d16967\") " pod="kube-system/kas-connection-checker-685895c7c9-zdvb2" Oct 30 21:00:02.397358 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.397280 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxsnl\" (UniqueName: \"kubernetes.io/projected/c0b2deee-2b0c-4c2c-9ac5-debd33f12ba3-kube-api-access-nxsnl\") pod \"kas-connection-checker-685895c7c9-b9fhn\" (UID: \"c0b2deee-2b0c-4c2c-9ac5-debd33f12ba3\") " pod="kube-system/kas-connection-checker-685895c7c9-b9fhn" Oct 30 21:00:02.397358 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.397313 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b61b643-c3cf-453c-875e-25576445ad4f-trusted-ca\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.397358 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.397332 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2749\" (UniqueName: \"kubernetes.io/projected/0b77d4c7-37eb-4bad-8367-31346289726e-kube-api-access-l2749\") pod \"cluster-monitoring-operator-884576c57-v5rrz\" (UID: \"0b77d4c7-37eb-4bad-8367-31346289726e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:02.397474 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.397387 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wbrhm\" (UID: \"9ffe8e19-6888-4eca-ae9d-9d0fc05ba752\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" Oct 30 21:00:02.397474 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.397449 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b61b643-c3cf-453c-875e-25576445ad4f-installation-pull-secrets\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.397474 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.397473 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ce43826-7550-4c62-8acc-48714541e426-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-frsk7\" (UID: \"4ce43826-7550-4c62-8acc-48714541e426\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7" Oct 30 21:00:02.397571 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.397491 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/705e2b0e-10e1-4710-8dfb-bbd862c8c084-config\") pod \"service-ca-operator-544d749c8c-vprbm\" (UID: \"705e2b0e-10e1-4710-8dfb-bbd862c8c084\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm" Oct 30 21:00:02.397571 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.397515 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:02.397571 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.397548 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-bound-sa-token\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.397744 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.397724 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmcw2\" (UniqueName: \"kubernetes.io/projected/6d9c7935-1d3d-4323-ac72-008a394e7f31-kube-api-access-nmcw2\") pod \"kas-connection-checker-685895c7c9-zr75n\" (UID: \"6d9c7935-1d3d-4323-ac72-008a394e7f31\") " pod="kube-system/kas-connection-checker-685895c7c9-zr75n" Oct 30 21:00:02.397800 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.397780 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-serving-cert\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:02.397850 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.397818 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgg8n\" (UniqueName: \"kubernetes.io/projected/3358db14-a01d-479e-a742-bdcd1ca80527-kube-api-access-cgg8n\") pod \"network-check-source-6f78d59f99-j5tcr\" (UID: \"3358db14-a01d-479e-a742-bdcd1ca80527\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j5tcr" Oct 30 21:00:02.397891 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.397855 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-stats-auth\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:02.397939 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.397909 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvl2l\" (UniqueName: \"kubernetes.io/projected/4ce43826-7550-4c62-8acc-48714541e426-kube-api-access-gvl2l\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-frsk7\" (UID: \"4ce43826-7550-4c62-8acc-48714541e426\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7" Oct 30 21:00:02.397993 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.397978 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0b77d4c7-37eb-4bad-8367-31346289726e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-v5rrz\" (UID: \"0b77d4c7-37eb-4bad-8367-31346289726e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:02.398052 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.398011 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-service-ca-bundle\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:02.398092 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.398055 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b61b643-c3cf-453c-875e-25576445ad4f-registry-certificates\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.398156 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.398120 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5r7xx\" (UID: \"ef2c01c8-12ef-4294-963e-4e840e2eecff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" Oct 30 21:00:02.398323 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.398293 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/705e2b0e-10e1-4710-8dfb-bbd862c8c084-config\") pod \"service-ca-operator-544d749c8c-vprbm\" (UID: \"705e2b0e-10e1-4710-8dfb-bbd862c8c084\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm" Oct 30 21:00:02.398391 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.398297 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-config\") pod \"console-operator-7cd56b8dbc-c5khs\" (UID: \"1a2fb9ac-a905-4e20-b1be-7928d5e7881a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:02.398391 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.398351 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4901add7-3312-4950-ae4c-4d6581380e9e-cert\") pod \"ingress-canary-mmzzs\" (UID: \"4901add7-3312-4950-ae4c-4d6581380e9e\") " pod="openshift-ingress-canary/ingress-canary-mmzzs" Oct 30 21:00:02.398505 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.398395 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b4ba52c-ad04-4a25-9158-89941367abd3-ca-trust-extracted\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.398505 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.398424 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-certificates\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.398505 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.398451 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v5rrz\" (UID: \"0b77d4c7-37eb-4bad-8367-31346289726e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:02.398505 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.398481 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-snapshots\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:02.398664 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.398628 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-service-ca-bundle\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:02.398664 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.398526 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:02.398762 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.398694 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhvqf\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-kube-api-access-rhvqf\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.398762 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.398721 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b67sr\" (UniqueName: \"kubernetes.io/projected/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-kube-api-access-b67sr\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:02.398762 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.398726 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:02.398762 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.398746 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.398926 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.398786 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrt6b\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-kube-api-access-vrt6b\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.398926 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.398888 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5r7xx\" (UID: \"ef2c01c8-12ef-4294-963e-4e840e2eecff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" Oct 30 21:00:02.399026 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.398923 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-snapshots\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:02.399026 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.398927 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6gqs\" (UniqueName: \"kubernetes.io/projected/705e2b0e-10e1-4710-8dfb-bbd862c8c084-kube-api-access-x6gqs\") pod \"service-ca-operator-544d749c8c-vprbm\" (UID: \"705e2b0e-10e1-4710-8dfb-bbd862c8c084\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm" Oct 30 21:00:02.399026 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.398965 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tf9r\" (UniqueName: \"kubernetes.io/projected/662ead00-b524-4a6e-815b-c3399b3c9e6c-kube-api-access-6tf9r\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:02.399026 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.398989 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:02.399026 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.399014 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w859\" (UniqueName: \"kubernetes.io/projected/4901add7-3312-4950-ae4c-4d6581380e9e-kube-api-access-4w859\") pod \"ingress-canary-mmzzs\" (UID: \"4901add7-3312-4950-ae4c-4d6581380e9e\") " pod="openshift-ingress-canary/ingress-canary-mmzzs" Oct 30 21:00:02.399203 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.399060 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-default-certificate\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:02.399203 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.399089 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwf67\" (UniqueName: \"kubernetes.io/projected/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-kube-api-access-pwf67\") pod \"console-operator-7cd56b8dbc-c5khs\" (UID: \"1a2fb9ac-a905-4e20-b1be-7928d5e7881a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:02.399203 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.399116 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ce43826-7550-4c62-8acc-48714541e426-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-frsk7\" (UID: \"4ce43826-7550-4c62-8acc-48714541e426\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7" Oct 30 21:00:02.399203 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.399175 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b61b643-c3cf-453c-875e-25576445ad4f-image-registry-private-configuration\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.399330 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.399229 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.399330 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.399285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:02.401060 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.400951 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/705e2b0e-10e1-4710-8dfb-bbd862c8c084-serving-cert\") pod \"service-ca-operator-544d749c8c-vprbm\" (UID: \"705e2b0e-10e1-4710-8dfb-bbd862c8c084\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm" Oct 30 21:00:02.401130 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.400982 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-serving-cert\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:02.418497 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.418477 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hfkg\" (UniqueName: \"kubernetes.io/projected/ef2c01c8-12ef-4294-963e-4e840e2eecff-kube-api-access-9hfkg\") pod \"cluster-samples-operator-6ddb6c8b94-5r7xx\" (UID: \"ef2c01c8-12ef-4294-963e-4e840e2eecff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" Oct 30 21:00:02.418647 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.418625 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2k65\" (UniqueName: \"kubernetes.io/projected/0f0b2878-417e-4986-8067-b50a16d16967-kube-api-access-z2k65\") pod \"kas-connection-checker-685895c7c9-zdvb2\" (UID: \"0f0b2878-417e-4986-8067-b50a16d16967\") " pod="kube-system/kas-connection-checker-685895c7c9-zdvb2" Oct 30 21:00:02.418708 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.418657 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2mms\" (UniqueName: \"kubernetes.io/projected/a66e14f7-3c72-48a5-88e8-e9f78a7c7ff5-kube-api-access-h2mms\") pod \"volume-data-source-validator-58c9f5f46d-87c42\" (UID: \"a66e14f7-3c72-48a5-88e8-e9f78a7c7ff5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87c42" Oct 30 21:00:02.418767 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.418744 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmcw2\" (UniqueName: \"kubernetes.io/projected/6d9c7935-1d3d-4323-ac72-008a394e7f31-kube-api-access-nmcw2\") pod \"kas-connection-checker-685895c7c9-zr75n\" (UID: \"6d9c7935-1d3d-4323-ac72-008a394e7f31\") " pod="kube-system/kas-connection-checker-685895c7c9-zr75n" Oct 30 21:00:02.419575 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.419549 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxsnl\" (UniqueName: \"kubernetes.io/projected/c0b2deee-2b0c-4c2c-9ac5-debd33f12ba3-kube-api-access-nxsnl\") pod \"kas-connection-checker-685895c7c9-b9fhn\" (UID: \"c0b2deee-2b0c-4c2c-9ac5-debd33f12ba3\") " pod="kube-system/kas-connection-checker-685895c7c9-b9fhn" Oct 30 21:00:02.419709 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.419695 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6gqs\" (UniqueName: \"kubernetes.io/projected/705e2b0e-10e1-4710-8dfb-bbd862c8c084-kube-api-access-x6gqs\") pod \"service-ca-operator-544d749c8c-vprbm\" (UID: \"705e2b0e-10e1-4710-8dfb-bbd862c8c084\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm" Oct 30 21:00:02.419842 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.419826 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b67sr\" (UniqueName: \"kubernetes.io/projected/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-kube-api-access-b67sr\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:02.500170 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500109 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2749\" (UniqueName: \"kubernetes.io/projected/0b77d4c7-37eb-4bad-8367-31346289726e-kube-api-access-l2749\") pod \"cluster-monitoring-operator-884576c57-v5rrz\" (UID: \"0b77d4c7-37eb-4bad-8367-31346289726e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:02.500170 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500141 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wbrhm\" (UID: \"9ffe8e19-6888-4eca-ae9d-9d0fc05ba752\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" Oct 30 21:00:02.500331 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500183 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b61b643-c3cf-453c-875e-25576445ad4f-installation-pull-secrets\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.500331 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500205 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ce43826-7550-4c62-8acc-48714541e426-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-frsk7\" (UID: \"4ce43826-7550-4c62-8acc-48714541e426\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7" Oct 30 21:00:02.500331 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500231 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-bound-sa-token\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.500467 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500417 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgg8n\" (UniqueName: \"kubernetes.io/projected/3358db14-a01d-479e-a742-bdcd1ca80527-kube-api-access-cgg8n\") pod \"network-check-source-6f78d59f99-j5tcr\" (UID: \"3358db14-a01d-479e-a742-bdcd1ca80527\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j5tcr" Oct 30 21:00:02.500467 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500456 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-stats-auth\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:02.500554 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500486 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvl2l\" (UniqueName: \"kubernetes.io/projected/4ce43826-7550-4c62-8acc-48714541e426-kube-api-access-gvl2l\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-frsk7\" (UID: \"4ce43826-7550-4c62-8acc-48714541e426\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7" Oct 30 21:00:02.500554 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500516 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0b77d4c7-37eb-4bad-8367-31346289726e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-v5rrz\" (UID: \"0b77d4c7-37eb-4bad-8367-31346289726e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:02.500554 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500543 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b61b643-c3cf-453c-875e-25576445ad4f-registry-certificates\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.500704 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500571 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-config\") pod \"console-operator-7cd56b8dbc-c5khs\" (UID: \"1a2fb9ac-a905-4e20-b1be-7928d5e7881a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:02.500704 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500597 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4901add7-3312-4950-ae4c-4d6581380e9e-cert\") pod \"ingress-canary-mmzzs\" (UID: \"4901add7-3312-4950-ae4c-4d6581380e9e\") " pod="openshift-ingress-canary/ingress-canary-mmzzs" Oct 30 21:00:02.500704 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500645 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b4ba52c-ad04-4a25-9158-89941367abd3-ca-trust-extracted\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.500704 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500671 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-certificates\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.500888 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500709 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v5rrz\" (UID: \"0b77d4c7-37eb-4bad-8367-31346289726e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:02.500888 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500738 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:02.500888 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500773 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhvqf\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-kube-api-access-rhvqf\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.500888 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500803 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.500888 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500806 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ce43826-7550-4c62-8acc-48714541e426-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-frsk7\" (UID: \"4ce43826-7550-4c62-8acc-48714541e426\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7" Oct 30 21:00:02.500888 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500826 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrt6b\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-kube-api-access-vrt6b\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.500888 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500854 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tf9r\" (UniqueName: \"kubernetes.io/projected/662ead00-b524-4a6e-815b-c3399b3c9e6c-kube-api-access-6tf9r\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:02.500888 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500877 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:02.501264 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500900 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w859\" (UniqueName: \"kubernetes.io/projected/4901add7-3312-4950-ae4c-4d6581380e9e-kube-api-access-4w859\") pod \"ingress-canary-mmzzs\" (UID: \"4901add7-3312-4950-ae4c-4d6581380e9e\") " pod="openshift-ingress-canary/ingress-canary-mmzzs" Oct 30 21:00:02.501264 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500904 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wbrhm\" (UID: \"9ffe8e19-6888-4eca-ae9d-9d0fc05ba752\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" Oct 30 21:00:02.501264 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500938 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-default-certificate\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:02.501264 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500964 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwf67\" (UniqueName: \"kubernetes.io/projected/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-kube-api-access-pwf67\") pod \"console-operator-7cd56b8dbc-c5khs\" (UID: \"1a2fb9ac-a905-4e20-b1be-7928d5e7881a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:02.501264 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.500989 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ce43826-7550-4c62-8acc-48714541e426-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-frsk7\" (UID: \"4ce43826-7550-4c62-8acc-48714541e426\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7" Oct 30 21:00:02.501264 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.501014 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b61b643-c3cf-453c-875e-25576445ad4f-image-registry-private-configuration\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.501264 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.501063 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.501264 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.501091 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:02.501264 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.501123 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wbrhm\" (UID: \"9ffe8e19-6888-4eca-ae9d-9d0fc05ba752\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" Oct 30 21:00:02.501264 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.501152 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz752\" (UniqueName: \"kubernetes.io/projected/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-kube-api-access-cz752\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:02.501264 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.501176 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b61b643-c3cf-453c-875e-25576445ad4f-ca-trust-extracted\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.501264 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.501199 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-bound-sa-token\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.501264 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.501223 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b61b643-c3cf-453c-875e-25576445ad4f-registry-certificates\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.501264 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.501242 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/662ead00-b524-4a6e-815b-c3399b3c9e6c-tmp-dir\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:02.501264 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.501255 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b4ba52c-ad04-4a25-9158-89941367abd3-ca-trust-extracted\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.501264 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.501268 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-serving-cert\") pod \"console-operator-7cd56b8dbc-c5khs\" (UID: \"1a2fb9ac-a905-4e20-b1be-7928d5e7881a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:02.501987 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:02.501340 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:00:02.501987 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:02.501353 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758fcd4795-j6bts: secret "image-registry-tls" not found Oct 30 21:00:02.501987 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:02.501410 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls podName:6b4ba52c-ad04-4a25-9158-89941367abd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:03.001394282 +0000 UTC m=+31.118325481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls") pod "image-registry-758fcd4795-j6bts" (UID: "6b4ba52c-ad04-4a25-9158-89941367abd3") : secret "image-registry-tls" not found Oct 30 21:00:02.501987 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.501594 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0b77d4c7-37eb-4bad-8367-31346289726e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-v5rrz\" (UID: \"0b77d4c7-37eb-4bad-8367-31346289726e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:02.501987 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:02.501638 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:00:02.501987 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:02.501695 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert podName:9ffe8e19-6888-4eca-ae9d-9d0fc05ba752 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:03.001680755 +0000 UTC m=+31.118611937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wbrhm" (UID: "9ffe8e19-6888-4eca-ae9d-9d0fc05ba752") : secret "networking-console-plugin-cert" not found Oct 30 21:00:02.501987 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.501736 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-certificates\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.502358 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:02.501997 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle podName:276d4076-cd5a-45d6-b4f8-2de14f9c9a0d nodeName:}" failed. No retries permitted until 2026-10-30 21:00:03.00198103 +0000 UTC m=+31.118912223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle") pod "router-default-6789dbc4b5-5gqgl" (UID: "276d4076-cd5a-45d6-b4f8-2de14f9c9a0d") : configmap references non-existent config key: service-ca.crt Oct 30 21:00:02.502358 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:02.502121 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:00:02.502358 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:02.502137 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56886949f8-xwhft: secret "image-registry-tls" not found Oct 30 21:00:02.502358 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.502167 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-config\") pod \"console-operator-7cd56b8dbc-c5khs\" (UID: \"1a2fb9ac-a905-4e20-b1be-7928d5e7881a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:02.502358 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:02.502270 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls podName:9b61b643-c3cf-453c-875e-25576445ad4f nodeName:}" failed. No retries permitted until 2026-10-30 21:00:03.002253245 +0000 UTC m=+31.119184428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls") pod "image-registry-56886949f8-xwhft" (UID: "9b61b643-c3cf-453c-875e-25576445ad4f") : secret "image-registry-tls" not found Oct 30 21:00:02.502358 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:02.502356 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:00:02.502632 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:02.502411 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:00:02.502632 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:02.502467 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs podName:276d4076-cd5a-45d6-b4f8-2de14f9c9a0d nodeName:}" failed. No retries permitted until 2026-10-30 21:00:03.002381949 +0000 UTC m=+31.119313128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs") pod "router-default-6789dbc4b5-5gqgl" (UID: "276d4076-cd5a-45d6-b4f8-2de14f9c9a0d") : secret "router-metrics-certs-default" not found Oct 30 21:00:02.502632 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.502487 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b4ba52c-ad04-4a25-9158-89941367abd3-image-registry-private-configuration\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.502632 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:02.502498 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:00:02.502632 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:02.502535 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls podName:0b77d4c7-37eb-4bad-8367-31346289726e nodeName:}" failed. No retries permitted until 2026-10-30 21:00:03.002521608 +0000 UTC m=+31.119452800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-v5rrz" (UID: "0b77d4c7-37eb-4bad-8367-31346289726e") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:00:02.502632 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.502536 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b4ba52c-ad04-4a25-9158-89941367abd3-installation-pull-secrets\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.502898 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:02.502741 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls podName:662ead00-b524-4a6e-815b-c3399b3c9e6c nodeName:}" failed. No retries permitted until 2026-10-30 21:00:03.002721342 +0000 UTC m=+31.119652535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls") pod "dns-default-888xx" (UID: "662ead00-b524-4a6e-815b-c3399b3c9e6c") : secret "dns-default-metrics-tls" not found Oct 30 21:00:02.503130 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.503071 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/662ead00-b524-4a6e-815b-c3399b3c9e6c-tmp-dir\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:02.503130 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.503103 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b61b643-c3cf-453c-875e-25576445ad4f-ca-trust-extracted\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.503130 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.503116 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/662ead00-b524-4a6e-815b-c3399b3c9e6c-config-volume\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:02.503298 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.503146 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b4ba52c-ad04-4a25-9158-89941367abd3-trusted-ca\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.503298 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.503168 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-trusted-ca\") pod \"console-operator-7cd56b8dbc-c5khs\" (UID: \"1a2fb9ac-a905-4e20-b1be-7928d5e7881a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:02.503298 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.503208 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b61b643-c3cf-453c-875e-25576445ad4f-trusted-ca\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.503629 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.503597 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b61b643-c3cf-453c-875e-25576445ad4f-installation-pull-secrets\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.504022 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.504001 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-stats-auth\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:02.504517 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.504490 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-trusted-ca\") pod \"console-operator-7cd56b8dbc-c5khs\" (UID: \"1a2fb9ac-a905-4e20-b1be-7928d5e7881a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:02.504602 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.504538 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b4ba52c-ad04-4a25-9158-89941367abd3-trusted-ca\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.504662 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.504594 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b61b643-c3cf-453c-875e-25576445ad4f-image-registry-private-configuration\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.504755 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.504705 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b61b643-c3cf-453c-875e-25576445ad4f-trusted-ca\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.504903 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.504875 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/662ead00-b524-4a6e-815b-c3399b3c9e6c-config-volume\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:02.505216 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.505188 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b4ba52c-ad04-4a25-9158-89941367abd3-image-registry-private-configuration\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.505789 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.505749 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-serving-cert\") pod \"console-operator-7cd56b8dbc-c5khs\" (UID: \"1a2fb9ac-a905-4e20-b1be-7928d5e7881a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:02.506146 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.506124 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4901add7-3312-4950-ae4c-4d6581380e9e-cert\") pod \"ingress-canary-mmzzs\" (UID: \"4901add7-3312-4950-ae4c-4d6581380e9e\") " pod="openshift-ingress-canary/ingress-canary-mmzzs" Oct 30 21:00:02.506624 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.506609 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b4ba52c-ad04-4a25-9158-89941367abd3-installation-pull-secrets\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.506918 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.506896 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ce43826-7550-4c62-8acc-48714541e426-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-frsk7\" (UID: \"4ce43826-7550-4c62-8acc-48714541e426\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7" Oct 30 21:00:02.507300 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.507286 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-default-certificate\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:02.519305 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.519274 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgg8n\" (UniqueName: \"kubernetes.io/projected/3358db14-a01d-479e-a742-bdcd1ca80527-kube-api-access-cgg8n\") pod \"network-check-source-6f78d59f99-j5tcr\" (UID: \"3358db14-a01d-479e-a742-bdcd1ca80527\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j5tcr" Oct 30 21:00:02.520889 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.520864 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2749\" (UniqueName: \"kubernetes.io/projected/0b77d4c7-37eb-4bad-8367-31346289726e-kube-api-access-l2749\") pod \"cluster-monitoring-operator-884576c57-v5rrz\" (UID: \"0b77d4c7-37eb-4bad-8367-31346289726e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:02.521353 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.521313 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvl2l\" (UniqueName: \"kubernetes.io/projected/4ce43826-7550-4c62-8acc-48714541e426-kube-api-access-gvl2l\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-frsk7\" (UID: \"4ce43826-7550-4c62-8acc-48714541e426\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7" Oct 30 21:00:02.521457 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.521381 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrt6b\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-kube-api-access-vrt6b\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.521731 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.521692 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-bound-sa-token\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:02.522756 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.522733 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz752\" (UniqueName: \"kubernetes.io/projected/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-kube-api-access-cz752\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:02.523122 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.523099 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhvqf\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-kube-api-access-rhvqf\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.523421 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.523400 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-bound-sa-token\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:02.523487 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.523447 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tf9r\" (UniqueName: \"kubernetes.io/projected/662ead00-b524-4a6e-815b-c3399b3c9e6c-kube-api-access-6tf9r\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:02.523549 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.523533 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwf67\" (UniqueName: \"kubernetes.io/projected/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-kube-api-access-pwf67\") pod \"console-operator-7cd56b8dbc-c5khs\" (UID: \"1a2fb9ac-a905-4e20-b1be-7928d5e7881a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:02.523724 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.523709 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w859\" (UniqueName: \"kubernetes.io/projected/4901add7-3312-4950-ae4c-4d6581380e9e-kube-api-access-4w859\") pod \"ingress-canary-mmzzs\" (UID: \"4901add7-3312-4950-ae4c-4d6581380e9e\") " pod="openshift-ingress-canary/ingress-canary-mmzzs" Oct 30 21:00:02.529549 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.529532 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zdvb2" Oct 30 21:00:02.538634 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.538613 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-zr75n" Oct 30 21:00:02.549649 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.549624 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-b9fhn" Oct 30 21:00:02.561953 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.561929 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87c42" Oct 30 21:00:02.570496 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.570291 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm" Oct 30 21:00:02.575636 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.575319 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:02.614241 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.613940 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:02.641698 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.641458 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j5tcr" Oct 30 21:00:02.642109 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.642050 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7" Oct 30 21:00:02.654432 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.654241 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mmzzs" Oct 30 21:00:02.844290 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.844241 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zr75n"] Oct 30 21:00:02.849204 ip-10-0-142-3 kubenswrapper[2577]: W1030 21:00:02.849167 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d9c7935_1d3d_4323_ac72_008a394e7f31.slice/crio-e1e8b14579a11d4ed7ed03b3a11a0ce6ebdd17b3bc316392fe600bb918be48c9 WatchSource:0}: Error finding container e1e8b14579a11d4ed7ed03b3a11a0ce6ebdd17b3bc316392fe600bb918be48c9: Status 404 returned error can't find the container with id e1e8b14579a11d4ed7ed03b3a11a0ce6ebdd17b3bc316392fe600bb918be48c9 Oct 30 21:00:02.853171 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.852459 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-zdvb2"] Oct 30 21:00:02.873207 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.873180 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-b9fhn"] Oct 30 21:00:02.877952 ip-10-0-142-3 kubenswrapper[2577]: W1030 21:00:02.877915 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b2deee_2b0c_4c2c_9ac5_debd33f12ba3.slice/crio-1ba5a3b6a76258cbeaf8a5c8af034efbdeb40e73ef0fe0493356c3d6b056133f WatchSource:0}: Error finding container 1ba5a3b6a76258cbeaf8a5c8af034efbdeb40e73ef0fe0493356c3d6b056133f: Status 404 returned error can't find the container with id 1ba5a3b6a76258cbeaf8a5c8af034efbdeb40e73ef0fe0493356c3d6b056133f Oct 30 21:00:02.908372 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.908334 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5r7xx\" (UID: \"ef2c01c8-12ef-4294-963e-4e840e2eecff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" Oct 30 21:00:02.908572 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:02.908554 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:00:02.908643 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:02.908635 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-tls podName:ef2c01c8-12ef-4294-963e-4e840e2eecff nodeName:}" failed. No retries permitted until 2026-10-30 21:00:03.908609106 +0000 UTC m=+32.025540299 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5r7xx" (UID: "ef2c01c8-12ef-4294-963e-4e840e2eecff") : secret "samples-operator-tls" not found Oct 30 21:00:02.924923 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.924892 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87c42"] Oct 30 21:00:02.930724 ip-10-0-142-3 kubenswrapper[2577]: W1030 21:00:02.930691 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda66e14f7_3c72_48a5_88e8_e9f78a7c7ff5.slice/crio-e63373561db72954e6ed55d6d4a214d939a57c30e5f981211acb86b0b653ab7f WatchSource:0}: Error finding container e63373561db72954e6ed55d6d4a214d939a57c30e5f981211acb86b0b653ab7f: Status 404 returned error can't find the container with id e63373561db72954e6ed55d6d4a214d939a57c30e5f981211acb86b0b653ab7f Oct 30 21:00:02.951347 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.951264 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm"] Oct 30 21:00:02.960851 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.960830 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-c5khs"] Oct 30 21:00:02.960944 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:02.960879 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-m7m4c"] Oct 30 21:00:03.011668 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.011640 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:03.011990 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.011973 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:03.012253 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.012235 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wbrhm\" (UID: \"9ffe8e19-6888-4eca-ae9d-9d0fc05ba752\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" Oct 30 21:00:03.012579 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.012563 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v5rrz\" (UID: \"0b77d4c7-37eb-4bad-8367-31346289726e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:03.012775 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.012761 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:03.012966 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.012953 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:03.016751 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.013166 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:03.017081 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:03.011902 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:00:03.017206 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:03.017195 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56886949f8-xwhft: secret "image-registry-tls" not found Oct 30 21:00:03.017348 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:03.012185 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:00:03.017547 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:03.012401 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:00:03.017668 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:03.012722 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:00:03.017820 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:03.012910 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:00:03.017920 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:03.013125 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:00:03.018025 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:03.016976 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle podName:276d4076-cd5a-45d6-b4f8-2de14f9c9a0d nodeName:}" failed. No retries permitted until 2026-10-30 21:00:04.016960377 +0000 UTC m=+32.133891554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle") pod "router-default-6789dbc4b5-5gqgl" (UID: "276d4076-cd5a-45d6-b4f8-2de14f9c9a0d") : configmap references non-existent config key: service-ca.crt Oct 30 21:00:03.018165 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:03.018147 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758fcd4795-j6bts: secret "image-registry-tls" not found Oct 30 21:00:03.018273 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:03.018232 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls podName:9b61b643-c3cf-453c-875e-25576445ad4f nodeName:}" failed. No retries permitted until 2026-10-30 21:00:04.018218683 +0000 UTC m=+32.135149865 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls") pod "image-registry-56886949f8-xwhft" (UID: "9b61b643-c3cf-453c-875e-25576445ad4f") : secret "image-registry-tls" not found Oct 30 21:00:03.018345 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:03.018339 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs podName:276d4076-cd5a-45d6-b4f8-2de14f9c9a0d nodeName:}" failed. No retries permitted until 2026-10-30 21:00:04.018330559 +0000 UTC m=+32.135261740 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs") pod "router-default-6789dbc4b5-5gqgl" (UID: "276d4076-cd5a-45d6-b4f8-2de14f9c9a0d") : secret "router-metrics-certs-default" not found Oct 30 21:00:03.018473 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:03.018456 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert podName:9ffe8e19-6888-4eca-ae9d-9d0fc05ba752 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:04.018447429 +0000 UTC m=+32.135378606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wbrhm" (UID: "9ffe8e19-6888-4eca-ae9d-9d0fc05ba752") : secret "networking-console-plugin-cert" not found Oct 30 21:00:03.018546 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:03.018540 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls podName:0b77d4c7-37eb-4bad-8367-31346289726e nodeName:}" failed. No retries permitted until 2026-10-30 21:00:04.018530742 +0000 UTC m=+32.135461919 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-v5rrz" (UID: "0b77d4c7-37eb-4bad-8367-31346289726e") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:00:03.018628 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:03.018621 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls podName:662ead00-b524-4a6e-815b-c3399b3c9e6c nodeName:}" failed. No retries permitted until 2026-10-30 21:00:04.018614249 +0000 UTC m=+32.135545426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls") pod "dns-default-888xx" (UID: "662ead00-b524-4a6e-815b-c3399b3c9e6c") : secret "dns-default-metrics-tls" not found Oct 30 21:00:03.018764 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:03.018755 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls podName:6b4ba52c-ad04-4a25-9158-89941367abd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:04.018743663 +0000 UTC m=+32.135674843 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls") pod "image-registry-758fcd4795-j6bts" (UID: "6b4ba52c-ad04-4a25-9158-89941367abd3") : secret "image-registry-tls" not found Oct 30 21:00:03.024519 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.024084 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-j5tcr"] Oct 30 21:00:03.026379 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.025656 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mmzzs"] Oct 30 21:00:03.031493 ip-10-0-142-3 kubenswrapper[2577]: W1030 21:00:03.031470 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4901add7_3312_4950_ae4c_4d6581380e9e.slice/crio-81d54f22853849278d8b2c649c0d4a515ab479ecfeef0013099e3f3f5dc690c5 WatchSource:0}: Error finding container 81d54f22853849278d8b2c649c0d4a515ab479ecfeef0013099e3f3f5dc690c5: Status 404 returned error can't find the container with id 81d54f22853849278d8b2c649c0d4a515ab479ecfeef0013099e3f3f5dc690c5 Oct 30 21:00:03.031675 ip-10-0-142-3 kubenswrapper[2577]: W1030 21:00:03.031657 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3358db14_a01d_479e_a742_bdcd1ca80527.slice/crio-93d288e4555cd3e2cc0345c8afcd8156f019e9f1443044bb18e3060667bd1079 WatchSource:0}: Error finding container 93d288e4555cd3e2cc0345c8afcd8156f019e9f1443044bb18e3060667bd1079: Status 404 returned error can't find the container with id 93d288e4555cd3e2cc0345c8afcd8156f019e9f1443044bb18e3060667bd1079 Oct 30 21:00:03.054952 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.054924 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7"] Oct 30 21:00:03.058867 ip-10-0-142-3 kubenswrapper[2577]: W1030 21:00:03.058844 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ce43826_7550_4c62_8acc_48714541e426.slice/crio-fdbd72ddb43ee4578e925ba88e43c3fb79e43f8fa5709ce70d51039ec2502586 WatchSource:0}: Error finding container fdbd72ddb43ee4578e925ba88e43c3fb79e43f8fa5709ce70d51039ec2502586: Status 404 returned error can't find the container with id fdbd72ddb43ee4578e925ba88e43c3fb79e43f8fa5709ce70d51039ec2502586 Oct 30 21:00:03.367070 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.366966 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 21:00:03.367070 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.366999 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 21:00:03.367070 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.366966 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 21:00:03.382004 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.381981 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:00:03.382638 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.382014 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jljmv\"" Oct 30 21:00:03.382638 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.382062 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fz2bf\"" Oct 30 21:00:03.382638 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.382069 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-56hzc\"" Oct 30 21:00:03.382638 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.382101 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:00:03.589416 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.589376 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" event={"ID":"1a2fb9ac-a905-4e20-b1be-7928d5e7881a","Type":"ContainerStarted","Data":"f7dd4d31803010cf19be81325287be8a522d556865e760858b16bfe0f4439421"} Oct 30 21:00:03.591253 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.591190 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-m7m4c" event={"ID":"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6","Type":"ContainerStarted","Data":"d9daf948fdbabb2d3d1e892ca5c110c1242fbad1590c0fd045f06fb61391b932"} Oct 30 21:00:03.593268 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.593238 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mmzzs" event={"ID":"4901add7-3312-4950-ae4c-4d6581380e9e","Type":"ContainerStarted","Data":"81d54f22853849278d8b2c649c0d4a515ab479ecfeef0013099e3f3f5dc690c5"} Oct 30 21:00:03.594829 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.594760 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm" event={"ID":"705e2b0e-10e1-4710-8dfb-bbd862c8c084","Type":"ContainerStarted","Data":"ae4fa48a1d708a46447ff5ac1999bb738e3759a7b20711ec7e5b1472cc0a5b61"} Oct 30 21:00:03.597231 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.596737 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zr75n" event={"ID":"6d9c7935-1d3d-4323-ac72-008a394e7f31","Type":"ContainerStarted","Data":"903ffb3884b1cff47a8039b460f209501224b3e22339d5528a210472f82554a3"} Oct 30 21:00:03.597231 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.596780 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zr75n" event={"ID":"6d9c7935-1d3d-4323-ac72-008a394e7f31","Type":"ContainerStarted","Data":"e1e8b14579a11d4ed7ed03b3a11a0ce6ebdd17b3bc316392fe600bb918be48c9"} Oct 30 21:00:03.601238 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.601192 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87c42" event={"ID":"a66e14f7-3c72-48a5-88e8-e9f78a7c7ff5","Type":"ContainerStarted","Data":"e63373561db72954e6ed55d6d4a214d939a57c30e5f981211acb86b0b653ab7f"} Oct 30 21:00:03.604123 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.603625 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zdvb2" event={"ID":"0f0b2878-417e-4986-8067-b50a16d16967","Type":"ContainerStarted","Data":"acff05b2fe013b430170715fe3bea4c744671ff4fa53c1b06181c9db76d79100"} Oct 30 21:00:03.604123 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.603677 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-zdvb2" event={"ID":"0f0b2878-417e-4986-8067-b50a16d16967","Type":"ContainerStarted","Data":"d122169f6e92689276586e461c8d1c60c93d73f4f62ef18a38b79104e603723b"} Oct 30 21:00:03.605106 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.605020 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7" event={"ID":"4ce43826-7550-4c62-8acc-48714541e426","Type":"ContainerStarted","Data":"fdbd72ddb43ee4578e925ba88e43c3fb79e43f8fa5709ce70d51039ec2502586"} Oct 30 21:00:03.606388 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.606365 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j5tcr" event={"ID":"3358db14-a01d-479e-a742-bdcd1ca80527","Type":"ContainerStarted","Data":"93d288e4555cd3e2cc0345c8afcd8156f019e9f1443044bb18e3060667bd1079"} Oct 30 21:00:03.608731 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.608671 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-b9fhn" event={"ID":"c0b2deee-2b0c-4c2c-9ac5-debd33f12ba3","Type":"ContainerStarted","Data":"1ce7ed7cbca8450a34595d9c71205862fbfee30c38ab0ce0c3c76bf6b1c69000"} Oct 30 21:00:03.608731 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.608707 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-b9fhn" event={"ID":"c0b2deee-2b0c-4c2c-9ac5-debd33f12ba3","Type":"ContainerStarted","Data":"1ba5a3b6a76258cbeaf8a5c8af034efbdeb40e73ef0fe0493356c3d6b056133f"} Oct 30 21:00:03.673071 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.673009 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-zr75n" podStartSLOduration=274.672996223 podStartE2EDuration="4m34.672996223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:03.636154311 +0000 UTC m=+31.753085510" watchObservedRunningTime="2026-10-30 21:00:03.672996223 +0000 UTC m=+31.789927414" Oct 30 21:00:03.725455 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.725183 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-b9fhn" podStartSLOduration=274.725165385 podStartE2EDuration="4m34.725165385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:03.674234069 +0000 UTC m=+31.791165268" watchObservedRunningTime="2026-10-30 21:00:03.725165385 +0000 UTC m=+31.842096588" Oct 30 21:00:03.725621 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.725508 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-zdvb2" podStartSLOduration=274.725500411 podStartE2EDuration="4m34.725500411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:03.723927554 +0000 UTC m=+31.840858756" watchObservedRunningTime="2026-10-30 21:00:03.725500411 +0000 UTC m=+31.842431601" Oct 30 21:00:03.931493 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:03.931409 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5r7xx\" (UID: \"ef2c01c8-12ef-4294-963e-4e840e2eecff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" Oct 30 21:00:03.931652 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:03.931579 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:00:03.931652 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:03.931637 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-tls podName:ef2c01c8-12ef-4294-963e-4e840e2eecff nodeName:}" failed. No retries permitted until 2026-10-30 21:00:05.931618042 +0000 UTC m=+34.048549221 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5r7xx" (UID: "ef2c01c8-12ef-4294-963e-4e840e2eecff") : secret "samples-operator-tls" not found Oct 30 21:00:04.032899 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:04.032860 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:04.033087 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:04.032916 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:04.033087 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:04.032947 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wbrhm\" (UID: \"9ffe8e19-6888-4eca-ae9d-9d0fc05ba752\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" Oct 30 21:00:04.033213 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:04.033099 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v5rrz\" (UID: \"0b77d4c7-37eb-4bad-8367-31346289726e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:04.033213 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:04.033130 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:04.033213 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:04.033158 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:04.033213 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:04.033188 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:04.033393 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:04.033336 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle podName:276d4076-cd5a-45d6-b4f8-2de14f9c9a0d nodeName:}" failed. No retries permitted until 2026-10-30 21:00:06.033317974 +0000 UTC m=+34.150249170 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle") pod "router-default-6789dbc4b5-5gqgl" (UID: "276d4076-cd5a-45d6-b4f8-2de14f9c9a0d") : configmap references non-existent config key: service-ca.crt Oct 30 21:00:04.033754 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:04.033734 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:00:04.033816 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:04.033757 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56886949f8-xwhft: secret "image-registry-tls" not found Oct 30 21:00:04.033816 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:04.033803 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls podName:9b61b643-c3cf-453c-875e-25576445ad4f nodeName:}" failed. No retries permitted until 2026-10-30 21:00:06.033787414 +0000 UTC m=+34.150718606 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls") pod "image-registry-56886949f8-xwhft" (UID: "9b61b643-c3cf-453c-875e-25576445ad4f") : secret "image-registry-tls" not found Oct 30 21:00:04.033910 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:04.033881 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:00:04.033961 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:04.033917 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs podName:276d4076-cd5a-45d6-b4f8-2de14f9c9a0d nodeName:}" failed. No retries permitted until 2026-10-30 21:00:06.033906689 +0000 UTC m=+34.150837885 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs") pod "router-default-6789dbc4b5-5gqgl" (UID: "276d4076-cd5a-45d6-b4f8-2de14f9c9a0d") : secret "router-metrics-certs-default" not found Oct 30 21:00:04.034018 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:04.033999 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:00:04.034080 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:04.034048 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert podName:9ffe8e19-6888-4eca-ae9d-9d0fc05ba752 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:06.034020798 +0000 UTC m=+34.150951995 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wbrhm" (UID: "9ffe8e19-6888-4eca-ae9d-9d0fc05ba752") : secret "networking-console-plugin-cert" not found Oct 30 21:00:04.034132 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:04.034103 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:00:04.034183 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:04.034135 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls podName:0b77d4c7-37eb-4bad-8367-31346289726e nodeName:}" failed. No retries permitted until 2026-10-30 21:00:06.034122915 +0000 UTC m=+34.151054106 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-v5rrz" (UID: "0b77d4c7-37eb-4bad-8367-31346289726e") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:00:04.034239 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:04.034185 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:00:04.034239 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:04.034212 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls podName:662ead00-b524-4a6e-815b-c3399b3c9e6c nodeName:}" failed. No retries permitted until 2026-10-30 21:00:06.034203086 +0000 UTC m=+34.151134266 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls") pod "dns-default-888xx" (UID: "662ead00-b524-4a6e-815b-c3399b3c9e6c") : secret "dns-default-metrics-tls" not found Oct 30 21:00:04.034331 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:04.034267 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:00:04.034331 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:04.034277 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758fcd4795-j6bts: secret "image-registry-tls" not found Oct 30 21:00:04.034331 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:04.034304 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls podName:6b4ba52c-ad04-4a25-9158-89941367abd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:06.034295157 +0000 UTC m=+34.151226349 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls") pod "image-registry-758fcd4795-j6bts" (UID: "6b4ba52c-ad04-4a25-9158-89941367abd3") : secret "image-registry-tls" not found Oct 30 21:00:04.620219 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:04.620125 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87c42" event={"ID":"a66e14f7-3c72-48a5-88e8-e9f78a7c7ff5","Type":"ContainerStarted","Data":"836cb1a3496c562800fcd8b57eea584ec182a8affb45edd463fbdc38a84d5fa7"} Oct 30 21:00:04.659256 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:04.658996 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87c42" podStartSLOduration=224.284966358 podStartE2EDuration="3m45.658975759s" totalImagesPullingTime="1.374009401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:04.657981187 +0000 UTC m=+32.774912403" watchObservedRunningTime="2026-10-30 21:00:04.658975759 +0000 UTC m=+32.775906958" Oct 30 21:00:05.957136 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:05.957016 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5r7xx\" (UID: \"ef2c01c8-12ef-4294-963e-4e840e2eecff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" Oct 30 21:00:05.958362 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:05.957688 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:00:05.958362 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:05.957784 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-tls podName:ef2c01c8-12ef-4294-963e-4e840e2eecff nodeName:}" failed. No retries permitted until 2026-10-30 21:00:09.957764143 +0000 UTC m=+38.074695323 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5r7xx" (UID: "ef2c01c8-12ef-4294-963e-4e840e2eecff") : secret "samples-operator-tls" not found Oct 30 21:00:06.061263 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:06.059849 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v5rrz\" (UID: \"0b77d4c7-37eb-4bad-8367-31346289726e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:06.061263 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:06.059901 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:06.061263 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:06.059932 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:06.061263 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:06.059965 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:06.061263 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:06.060000 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:06.061263 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:06.060041 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:06.061263 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:06.060072 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wbrhm\" (UID: \"9ffe8e19-6888-4eca-ae9d-9d0fc05ba752\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" Oct 30 21:00:06.061263 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:06.060209 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:00:06.061263 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:06.060276 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert podName:9ffe8e19-6888-4eca-ae9d-9d0fc05ba752 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:10.06025665 +0000 UTC m=+38.177187872 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wbrhm" (UID: "9ffe8e19-6888-4eca-ae9d-9d0fc05ba752") : secret "networking-console-plugin-cert" not found Oct 30 21:00:06.061263 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:06.060365 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:00:06.061263 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:06.060404 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls podName:0b77d4c7-37eb-4bad-8367-31346289726e nodeName:}" failed. No retries permitted until 2026-10-30 21:00:10.060392655 +0000 UTC m=+38.177323852 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-v5rrz" (UID: "0b77d4c7-37eb-4bad-8367-31346289726e") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:00:06.061263 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:06.060466 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:00:06.061263 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:06.060499 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls podName:662ead00-b524-4a6e-815b-c3399b3c9e6c nodeName:}" failed. No retries permitted until 2026-10-30 21:00:10.06048918 +0000 UTC m=+38.177420360 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls") pod "dns-default-888xx" (UID: "662ead00-b524-4a6e-815b-c3399b3c9e6c") : secret "dns-default-metrics-tls" not found Oct 30 21:00:06.061263 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:06.060565 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:00:06.061263 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:06.060577 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758fcd4795-j6bts: secret "image-registry-tls" not found Oct 30 21:00:06.061263 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:06.060610 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls podName:6b4ba52c-ad04-4a25-9158-89941367abd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:10.06060044 +0000 UTC m=+38.177531634 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls") pod "image-registry-758fcd4795-j6bts" (UID: "6b4ba52c-ad04-4a25-9158-89941367abd3") : secret "image-registry-tls" not found Oct 30 21:00:06.062164 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:06.060690 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle podName:276d4076-cd5a-45d6-b4f8-2de14f9c9a0d nodeName:}" failed. No retries permitted until 2026-10-30 21:00:10.060680126 +0000 UTC m=+38.177611317 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle") pod "router-default-6789dbc4b5-5gqgl" (UID: "276d4076-cd5a-45d6-b4f8-2de14f9c9a0d") : configmap references non-existent config key: service-ca.crt Oct 30 21:00:06.062164 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:06.060748 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:00:06.062164 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:06.060757 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56886949f8-xwhft: secret "image-registry-tls" not found Oct 30 21:00:06.062164 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:06.060782 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls podName:9b61b643-c3cf-453c-875e-25576445ad4f nodeName:}" failed. No retries permitted until 2026-10-30 21:00:10.060773717 +0000 UTC m=+38.177704905 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls") pod "image-registry-56886949f8-xwhft" (UID: "9b61b643-c3cf-453c-875e-25576445ad4f") : secret "image-registry-tls" not found Oct 30 21:00:06.062164 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:06.060836 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:00:06.062164 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:06.060866 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs podName:276d4076-cd5a-45d6-b4f8-2de14f9c9a0d nodeName:}" failed. No retries permitted until 2026-10-30 21:00:10.060855538 +0000 UTC m=+38.177786731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs") pod "router-default-6789dbc4b5-5gqgl" (UID: "276d4076-cd5a-45d6-b4f8-2de14f9c9a0d") : secret "router-metrics-certs-default" not found Oct 30 21:00:06.161246 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:06.161210 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs\") pod \"network-metrics-daemon-7d6gs\" (UID: \"f6337834-db43-4b8e-a955-9691f72e0f19\") " pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 21:00:06.161431 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:06.161296 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 21:00:06.161431 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:06.161409 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:00:06.161542 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:06.161480 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs podName:f6337834-db43-4b8e-a955-9691f72e0f19 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:38.161460969 +0000 UTC m=+66.278392164 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs") pod "network-metrics-daemon-7d6gs" (UID: "f6337834-db43-4b8e-a955-9691f72e0f19") : secret "metrics-daemon-secret" not found Oct 30 21:00:06.181432 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:06.181404 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 21:00:06.262556 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:06.262491 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz6tt\" (UniqueName: \"kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt\") pod \"network-check-target-mfgjb\" (UID: \"4e46662c-bfe6-45f8-8a23-89ea5fc92ae5\") " pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 21:00:06.266386 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:06.266354 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz6tt\" (UniqueName: \"kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt\") pod \"network-check-target-mfgjb\" (UID: \"4e46662c-bfe6-45f8-8a23-89ea5fc92ae5\") " pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 21:00:06.379966 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:06.379898 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 21:00:06.384124 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:06.384096 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 21:00:06.700792 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:06.700756 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m9dm5"] Oct 30 21:00:06.706918 ip-10-0-142-3 kubenswrapper[2577]: W1030 21:00:06.706888 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae2e7bf_9e4d_4839_acfc_e2e3ca66df0e.slice/crio-4307f887bef1e1ad6367874facafc8cffd4c890d6415ff5c3dd10da77ae5b8f9 WatchSource:0}: Error finding container 4307f887bef1e1ad6367874facafc8cffd4c890d6415ff5c3dd10da77ae5b8f9: Status 404 returned error can't find the container with id 4307f887bef1e1ad6367874facafc8cffd4c890d6415ff5c3dd10da77ae5b8f9 Oct 30 21:00:06.743334 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:06.743227 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mfgjb"] Oct 30 21:00:06.754308 ip-10-0-142-3 kubenswrapper[2577]: W1030 21:00:06.754268 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e46662c_bfe6_45f8_8a23_89ea5fc92ae5.slice/crio-635e766e79e31aed78f48ee33e9c38219337c8af31bced336183205b22c540ad WatchSource:0}: Error finding container 635e766e79e31aed78f48ee33e9c38219337c8af31bced336183205b22c540ad: Status 404 returned error can't find the container with id 635e766e79e31aed78f48ee33e9c38219337c8af31bced336183205b22c540ad Oct 30 21:00:07.635383 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:07.635098 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mfgjb" event={"ID":"4e46662c-bfe6-45f8-8a23-89ea5fc92ae5","Type":"ContainerStarted","Data":"635e766e79e31aed78f48ee33e9c38219337c8af31bced336183205b22c540ad"} Oct 30 21:00:07.637225 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:07.637167 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m9dm5" event={"ID":"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e","Type":"ContainerStarted","Data":"4307f887bef1e1ad6367874facafc8cffd4c890d6415ff5c3dd10da77ae5b8f9"} Oct 30 21:00:07.639423 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:07.639398 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/0.log" Oct 30 21:00:07.639535 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:07.639435 2577 generic.go:356] "Generic (PLEG): container finished" podID="1a2fb9ac-a905-4e20-b1be-7928d5e7881a" containerID="29b19be9f50c86224b2043223027cfd91195860c59cba5ceef3d7b7d2cb3edff" exitCode=255 Oct 30 21:00:07.639535 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:07.639497 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" event={"ID":"1a2fb9ac-a905-4e20-b1be-7928d5e7881a","Type":"ContainerDied","Data":"29b19be9f50c86224b2043223027cfd91195860c59cba5ceef3d7b7d2cb3edff"} Oct 30 21:00:07.639801 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:07.639784 2577 scope.go:122] "RemoveContainer" containerID="29b19be9f50c86224b2043223027cfd91195860c59cba5ceef3d7b7d2cb3edff" Oct 30 21:00:08.651457 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.651405 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mmzzs" event={"ID":"4901add7-3312-4950-ae4c-4d6581380e9e","Type":"ContainerStarted","Data":"0d198f2c402953d39968042c2810881ec950cb2e10b5afe9348edf17abf4df66"} Oct 30 21:00:08.656220 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.656186 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm" event={"ID":"705e2b0e-10e1-4710-8dfb-bbd862c8c084","Type":"ContainerStarted","Data":"a3f9d7ad7953e493dcad1e9a8cb28b83bf31a3796b2aa6f324b2572a42ece688"} Oct 30 21:00:08.663199 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.663168 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-q9bnp"] Oct 30 21:00:08.675409 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.673069 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:00:08.675409 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.673612 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/0.log" Oct 30 21:00:08.675409 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.673646 2577 generic.go:356] "Generic (PLEG): container finished" podID="1a2fb9ac-a905-4e20-b1be-7928d5e7881a" containerID="cb966d4ceaaab215d2d47172a0b5244daddd96c1007b782eaaa1ed2d0d47b7ee" exitCode=255 Oct 30 21:00:08.675409 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.673947 2577 scope.go:122] "RemoveContainer" containerID="cb966d4ceaaab215d2d47172a0b5244daddd96c1007b782eaaa1ed2d0d47b7ee" Oct 30 21:00:08.675409 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:08.674149 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-c5khs_openshift-console-operator(1a2fb9ac-a905-4e20-b1be-7928d5e7881a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" podUID="1a2fb9ac-a905-4e20-b1be-7928d5e7881a" Oct 30 21:00:08.706412 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.706347 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q9bnp"] Oct 30 21:00:08.706412 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.706405 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7" event={"ID":"4ce43826-7550-4c62-8acc-48714541e426","Type":"ContainerStarted","Data":"0de05589dcb53d97bf066d86d8415cbe1973c30429448da0a7c744f7c6553995"} Oct 30 21:00:08.706929 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.706432 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j5tcr" event={"ID":"3358db14-a01d-479e-a742-bdcd1ca80527","Type":"ContainerStarted","Data":"2634bdba0b85a8b63a628ebb8c819285bac9bc674f2879ededede6812a060b77"} Oct 30 21:00:08.706929 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.706452 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" event={"ID":"1a2fb9ac-a905-4e20-b1be-7928d5e7881a","Type":"ContainerDied","Data":"cb966d4ceaaab215d2d47172a0b5244daddd96c1007b782eaaa1ed2d0d47b7ee"} Oct 30 21:00:08.706929 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.706476 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-m7m4c" event={"ID":"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6","Type":"ContainerStarted","Data":"14e178e155cb5c8f968b53dd6215ca77ba5047023842b774dab62b80f5246ba1"} Oct 30 21:00:08.706929 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.706571 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:08.706929 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.706626 2577 scope.go:122] "RemoveContainer" containerID="29b19be9f50c86224b2043223027cfd91195860c59cba5ceef3d7b7d2cb3edff" Oct 30 21:00:08.736330 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.735543 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm" podStartSLOduration=224.853584969 podStartE2EDuration="3m49.735523274s" totalImagesPullingTime="4.881938305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:08.735286026 +0000 UTC m=+36.852217439" watchObservedRunningTime="2026-10-30 21:00:08.735523274 +0000 UTC m=+36.852454473" Oct 30 21:00:08.736330 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.735769 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mmzzs" podStartSLOduration=1.346839839 podStartE2EDuration="6.735762765s" totalImagesPullingTime="5.388922926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:08.691672866 +0000 UTC m=+36.808604066" watchObservedRunningTime="2026-10-30 21:00:08.735762765 +0000 UTC m=+36.852693965" Oct 30 21:00:08.797625 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.797594 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:00:08.797783 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.797690 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l57v4\"" Oct 30 21:00:08.797783 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.797749 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:00:08.875859 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.875793 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-m7m4c" podStartSLOduration=225.013462042 podStartE2EDuration="3m49.875774104s" totalImagesPullingTime="4.862312062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:08.832428323 +0000 UTC m=+36.949359519" watchObservedRunningTime="2026-10-30 21:00:08.875774104 +0000 UTC m=+36.992705304" Oct 30 21:00:08.876046 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.875915 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7" podStartSLOduration=226.099613802 podStartE2EDuration="3m50.875908236s" totalImagesPullingTime="4.776294434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:08.874062025 +0000 UTC m=+36.990993230" watchObservedRunningTime="2026-10-30 21:00:08.875908236 +0000 UTC m=+36.992839436" Oct 30 21:00:08.896263 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.896233 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4ef3fe7-a7fb-473c-a301-9073d502a86a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:08.896420 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.896402 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:08.896496 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.896479 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4ef3fe7-a7fb-473c-a301-9073d502a86a-crio-socket\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:08.896601 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.896577 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4ef3fe7-a7fb-473c-a301-9073d502a86a-data-volume\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:08.896665 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.896622 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9bws\" (UniqueName: \"kubernetes.io/projected/b4ef3fe7-a7fb-473c-a301-9073d502a86a-kube-api-access-r9bws\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:08.913257 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.913208 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j5tcr" podStartSLOduration=223.504441445 podStartE2EDuration="3m48.913195988s" totalImagesPullingTime="5.408754543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:08.910918732 +0000 UTC m=+37.027849932" watchObservedRunningTime="2026-10-30 21:00:08.913195988 +0000 UTC m=+37.030127187" Oct 30 21:00:08.997478 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.997392 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4ef3fe7-a7fb-473c-a301-9073d502a86a-crio-socket\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:08.997478 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.997446 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4ef3fe7-a7fb-473c-a301-9073d502a86a-data-volume\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:08.997478 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.997472 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9bws\" (UniqueName: \"kubernetes.io/projected/b4ef3fe7-a7fb-473c-a301-9073d502a86a-kube-api-access-r9bws\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:08.997723 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.997589 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4ef3fe7-a7fb-473c-a301-9073d502a86a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:08.997723 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.997683 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:08.997823 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:08.997806 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:00:08.998263 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:08.997880 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls podName:b4ef3fe7-a7fb-473c-a301-9073d502a86a nodeName:}" failed. No retries permitted until 2026-10-30 21:00:09.497860787 +0000 UTC m=+37.614791978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q9bnp" (UID: "b4ef3fe7-a7fb-473c-a301-9073d502a86a") : secret "insights-runtime-extractor-tls" not found Oct 30 21:00:08.998263 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.998163 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4ef3fe7-a7fb-473c-a301-9073d502a86a-crio-socket\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:08.998263 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:08.998209 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:00:08.998263 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:08.998214 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4ef3fe7-a7fb-473c-a301-9073d502a86a-data-volume\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:08.998263 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:08.998267 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4ef3fe7-a7fb-473c-a301-9073d502a86a-kube-rbac-proxy-cm podName:b4ef3fe7-a7fb-473c-a301-9073d502a86a nodeName:}" failed. No retries permitted until 2026-10-30 21:00:09.498250058 +0000 UTC m=+37.615181251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b4ef3fe7-a7fb-473c-a301-9073d502a86a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q9bnp" (UID: "b4ef3fe7-a7fb-473c-a301-9073d502a86a") : configmap "kube-rbac-proxy" not found Oct 30 21:00:09.030138 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:09.028433 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9bws\" (UniqueName: \"kubernetes.io/projected/b4ef3fe7-a7fb-473c-a301-9073d502a86a-kube-api-access-r9bws\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:09.093674 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:09.093639 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5v597"] Oct 30 21:00:09.101906 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:09.101875 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5v597" Oct 30 21:00:09.102885 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:09.102855 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qzp2\" (UniqueName: \"kubernetes.io/projected/c790e080-b546-409c-8ed2-1fc23982984b-kube-api-access-6qzp2\") pod \"migrator-66df5d496b-5v597\" (UID: \"c790e080-b546-409c-8ed2-1fc23982984b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5v597" Oct 30 21:00:09.119363 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:09.119291 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5v597"] Oct 30 21:00:09.204286 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:09.204247 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qzp2\" (UniqueName: \"kubernetes.io/projected/c790e080-b546-409c-8ed2-1fc23982984b-kube-api-access-6qzp2\") pod \"migrator-66df5d496b-5v597\" (UID: \"c790e080-b546-409c-8ed2-1fc23982984b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5v597" Oct 30 21:00:09.215718 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:09.215688 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:00:09.215718 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:09.215688 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:00:09.215945 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:09.215776 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mttl5\"" Oct 30 21:00:09.224225 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:09.224199 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qzp2\" (UniqueName: \"kubernetes.io/projected/c790e080-b546-409c-8ed2-1fc23982984b-kube-api-access-6qzp2\") pod \"migrator-66df5d496b-5v597\" (UID: \"c790e080-b546-409c-8ed2-1fc23982984b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5v597" Oct 30 21:00:09.421832 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:09.421797 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5v597" Oct 30 21:00:09.506458 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:09.506425 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:09.506589 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:09.506534 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4ef3fe7-a7fb-473c-a301-9073d502a86a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:09.506639 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:09.506623 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:00:09.506882 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:09.506870 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4ef3fe7-a7fb-473c-a301-9073d502a86a-kube-rbac-proxy-cm podName:b4ef3fe7-a7fb-473c-a301-9073d502a86a nodeName:}" failed. No retries permitted until 2026-10-30 21:00:10.506852319 +0000 UTC m=+38.623783519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b4ef3fe7-a7fb-473c-a301-9073d502a86a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q9bnp" (UID: "b4ef3fe7-a7fb-473c-a301-9073d502a86a") : configmap "kube-rbac-proxy" not found Oct 30 21:00:09.506930 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:09.506879 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:00:09.506961 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:09.506944 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls podName:b4ef3fe7-a7fb-473c-a301-9073d502a86a nodeName:}" failed. No retries permitted until 2026-10-30 21:00:10.506927913 +0000 UTC m=+38.623859105 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q9bnp" (UID: "b4ef3fe7-a7fb-473c-a301-9073d502a86a") : secret "insights-runtime-extractor-tls" not found Oct 30 21:00:09.683723 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:09.683648 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:00:09.684193 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:09.684052 2577 scope.go:122] "RemoveContainer" containerID="cb966d4ceaaab215d2d47172a0b5244daddd96c1007b782eaaa1ed2d0d47b7ee" Oct 30 21:00:09.684299 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:09.684273 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-c5khs_openshift-console-operator(1a2fb9ac-a905-4e20-b1be-7928d5e7881a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" podUID="1a2fb9ac-a905-4e20-b1be-7928d5e7881a" Oct 30 21:00:09.757104 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:09.757075 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-5v597"] Oct 30 21:00:09.761718 ip-10-0-142-3 kubenswrapper[2577]: W1030 21:00:09.761684 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc790e080_b546_409c_8ed2_1fc23982984b.slice/crio-5ae83504a960547e4f577294464b10264d64c8b625de5f99452e0f3ccda56293 WatchSource:0}: Error finding container 5ae83504a960547e4f577294464b10264d64c8b625de5f99452e0f3ccda56293: Status 404 returned error can't find the container with id 5ae83504a960547e4f577294464b10264d64c8b625de5f99452e0f3ccda56293 Oct 30 21:00:10.013496 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:10.013413 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5r7xx\" (UID: \"ef2c01c8-12ef-4294-963e-4e840e2eecff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" Oct 30 21:00:10.013662 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:10.013560 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 21:00:10.013662 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:10.013627 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-tls podName:ef2c01c8-12ef-4294-963e-4e840e2eecff nodeName:}" failed. No retries permitted until 2026-10-30 21:00:18.0136067 +0000 UTC m=+46.130537892 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-5r7xx" (UID: "ef2c01c8-12ef-4294-963e-4e840e2eecff") : secret "samples-operator-tls" not found Oct 30 21:00:10.114518 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:10.114482 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:10.114617 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:10.114531 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:10.114617 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:10.114562 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:10.114617 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:10.114595 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:10.114617 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:10.114618 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:10.114866 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:10.114648 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wbrhm\" (UID: \"9ffe8e19-6888-4eca-ae9d-9d0fc05ba752\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" Oct 30 21:00:10.114866 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:10.114769 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v5rrz\" (UID: \"0b77d4c7-37eb-4bad-8367-31346289726e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:10.114936 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:10.114899 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:00:10.114963 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:10.114948 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls podName:0b77d4c7-37eb-4bad-8367-31346289726e nodeName:}" failed. No retries permitted until 2026-10-30 21:00:18.11493471 +0000 UTC m=+46.231865887 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-v5rrz" (UID: "0b77d4c7-37eb-4bad-8367-31346289726e") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:00:10.115158 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:10.115142 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:00:10.115247 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:10.115160 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56886949f8-xwhft: secret "image-registry-tls" not found Oct 30 21:00:10.115247 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:10.115171 2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:00:10.115247 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:10.115205 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls podName:9b61b643-c3cf-453c-875e-25576445ad4f nodeName:}" failed. No retries permitted until 2026-10-30 21:00:18.1151899 +0000 UTC m=+46.232121093 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls") pod "image-registry-56886949f8-xwhft" (UID: "9b61b643-c3cf-453c-875e-25576445ad4f") : secret "image-registry-tls" not found Oct 30 21:00:10.115247 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:10.115233 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert podName:9ffe8e19-6888-4eca-ae9d-9d0fc05ba752 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:18.115216084 +0000 UTC m=+46.232147277 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wbrhm" (UID: "9ffe8e19-6888-4eca-ae9d-9d0fc05ba752") : secret "networking-console-plugin-cert" not found Oct 30 21:00:10.115247 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:10.115236 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:00:10.115438 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:10.115180 2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:00:10.115438 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:10.115275 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle podName:276d4076-cd5a-45d6-b4f8-2de14f9c9a0d nodeName:}" failed. No retries permitted until 2026-10-30 21:00:18.115268261 +0000 UTC m=+46.232199451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle") pod "router-default-6789dbc4b5-5gqgl" (UID: "276d4076-cd5a-45d6-b4f8-2de14f9c9a0d") : configmap references non-existent config key: service-ca.crt Oct 30 21:00:10.115438 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:10.115281 2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758fcd4795-j6bts: secret "image-registry-tls" not found Oct 30 21:00:10.115438 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:10.115284 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:00:10.115438 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:10.115288 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls podName:662ead00-b524-4a6e-815b-c3399b3c9e6c nodeName:}" failed. No retries permitted until 2026-10-30 21:00:18.115283257 +0000 UTC m=+46.232214434 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls") pod "dns-default-888xx" (UID: "662ead00-b524-4a6e-815b-c3399b3c9e6c") : secret "dns-default-metrics-tls" not found Oct 30 21:00:10.115438 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:10.115324 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls podName:6b4ba52c-ad04-4a25-9158-89941367abd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:18.115313862 +0000 UTC m=+46.232245042 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls") pod "image-registry-758fcd4795-j6bts" (UID: "6b4ba52c-ad04-4a25-9158-89941367abd3") : secret "image-registry-tls" not found Oct 30 21:00:10.115438 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:10.115338 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs podName:276d4076-cd5a-45d6-b4f8-2de14f9c9a0d nodeName:}" failed. No retries permitted until 2026-10-30 21:00:18.115329777 +0000 UTC m=+46.232260962 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs") pod "router-default-6789dbc4b5-5gqgl" (UID: "276d4076-cd5a-45d6-b4f8-2de14f9c9a0d") : secret "router-metrics-certs-default" not found Oct 30 21:00:10.518910 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:10.518875 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:10.519118 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:10.519000 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4ef3fe7-a7fb-473c-a301-9073d502a86a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:10.519118 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:10.519101 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:00:10.519241 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:10.519156 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4ef3fe7-a7fb-473c-a301-9073d502a86a-kube-rbac-proxy-cm podName:b4ef3fe7-a7fb-473c-a301-9073d502a86a nodeName:}" failed. No retries permitted until 2026-10-30 21:00:12.519143405 +0000 UTC m=+40.636074604 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b4ef3fe7-a7fb-473c-a301-9073d502a86a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q9bnp" (UID: "b4ef3fe7-a7fb-473c-a301-9073d502a86a") : configmap "kube-rbac-proxy" not found Oct 30 21:00:10.519456 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:10.519426 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:00:10.519672 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:10.519493 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls podName:b4ef3fe7-a7fb-473c-a301-9073d502a86a nodeName:}" failed. No retries permitted until 2026-10-30 21:00:12.519476179 +0000 UTC m=+40.636407371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q9bnp" (UID: "b4ef3fe7-a7fb-473c-a301-9073d502a86a") : secret "insights-runtime-extractor-tls" not found Oct 30 21:00:10.688772 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:10.688732 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mfgjb" event={"ID":"4e46662c-bfe6-45f8-8a23-89ea5fc92ae5","Type":"ContainerStarted","Data":"72e78a70f167e5e1492f3c36e91d9dc2f5a0f4acfd3c677ca83d5e5ba2e1245b"} Oct 30 21:00:10.689212 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:10.689017 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 21:00:10.690533 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:10.690510 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5v597" event={"ID":"c790e080-b546-409c-8ed2-1fc23982984b","Type":"ContainerStarted","Data":"5ae83504a960547e4f577294464b10264d64c8b625de5f99452e0f3ccda56293"} Oct 30 21:00:11.696059 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:11.695939 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5v597" event={"ID":"c790e080-b546-409c-8ed2-1fc23982984b","Type":"ContainerStarted","Data":"31dda6b2df4a43b4f8bdc32327fb655fca21af1d97684793cddaa2d95985fd92"} Oct 30 21:00:11.733713 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:11.733665 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mfgjb" podStartSLOduration=35.395761797 podStartE2EDuration="38.733649052s" totalImagesPullingTime="3.337887255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:10.727291151 +0000 UTC m=+38.844222352" watchObservedRunningTime="2026-10-30 21:00:11.733649052 +0000 UTC m=+39.850580252" Oct 30 21:00:11.734609 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:11.734574 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5v597" podStartSLOduration=1.197747345 podStartE2EDuration="2.734563995s" totalImagesPullingTime="1.53681665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:00:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:11.732701574 +0000 UTC m=+39.849632772" watchObservedRunningTime="2026-10-30 21:00:11.734563995 +0000 UTC m=+39.851495197" Oct 30 21:00:12.538369 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:12.538340 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4ef3fe7-a7fb-473c-a301-9073d502a86a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:12.538485 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:12.538426 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:12.538485 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:12.538465 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:00:12.538600 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:12.538532 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:00:12.538600 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:12.538538 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b4ef3fe7-a7fb-473c-a301-9073d502a86a-kube-rbac-proxy-cm podName:b4ef3fe7-a7fb-473c-a301-9073d502a86a nodeName:}" failed. No retries permitted until 2026-10-30 21:00:16.538519668 +0000 UTC m=+44.655450858 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b4ef3fe7-a7fb-473c-a301-9073d502a86a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-q9bnp" (UID: "b4ef3fe7-a7fb-473c-a301-9073d502a86a") : configmap "kube-rbac-proxy" not found Oct 30 21:00:12.538600 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:12.538570 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls podName:b4ef3fe7-a7fb-473c-a301-9073d502a86a nodeName:}" failed. No retries permitted until 2026-10-30 21:00:16.53856008 +0000 UTC m=+44.655491261 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q9bnp" (UID: "b4ef3fe7-a7fb-473c-a301-9073d502a86a") : secret "insights-runtime-extractor-tls" not found Oct 30 21:00:12.614502 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:12.614322 2577 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:12.614502 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:12.614366 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:12.614688 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:12.614676 2577 scope.go:122] "RemoveContainer" containerID="cb966d4ceaaab215d2d47172a0b5244daddd96c1007b782eaaa1ed2d0d47b7ee" Oct 30 21:00:12.614873 ip-10-0-142-3 kubenswrapper[2577]: E1030 21:00:12.614854 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-c5khs_openshift-console-operator(1a2fb9ac-a905-4e20-b1be-7928d5e7881a)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" podUID="1a2fb9ac-a905-4e20-b1be-7928d5e7881a" Oct 30 21:00:12.702967 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:12.702924 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m9dm5" event={"ID":"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e","Type":"ContainerStarted","Data":"2fdff8389258e9a198d5925d072dec8f0094f80ed49ed8d160479ac89e4b1946"} Oct 30 21:00:12.741726 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:12.741681 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-m9dm5" podStartSLOduration=33.942802418 podStartE2EDuration="39.74166509s" totalImagesPullingTime="5.798862672s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:59:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:12.741211009 +0000 UTC m=+40.858142209" watchObservedRunningTime="2026-10-30 21:00:12.74166509 +0000 UTC m=+40.858596292" Oct 30 21:00:12.809652 ip-10-0-142-3 kubenswrapper[2577]: I1030 21:00:12.809546 2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:00:12.809743 ip-10-0-142-3 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:00:12.826399 ip-10-0-142-3 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:00:12.826566 ip-10-0-142-3 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:00:12.826630 ip-10-0-142-3 systemd[1]: kubelet.service: Consumed 2.896s CPU time. Oct 30 21:00:12.842631 ip-10-0-142-3 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:00:12.932766 ip-10-0-142-3 kubenswrapper[4926]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:00:12.932766 ip-10-0-142-3 kubenswrapper[4926]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:00:12.932766 ip-10-0-142-3 kubenswrapper[4926]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:00:12.932766 ip-10-0-142-3 kubenswrapper[4926]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:00:12.935053 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.934948 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:00:12.935053 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935049 4926 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:00:12.935053 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935055 4926 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:00:12.935053 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935060 4926 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:00:12.935336 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935074 4926 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:00:12.935336 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935078 4926 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:00:12.935336 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935083 4926 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:00:12.935336 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935087 4926 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:00:12.935336 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935090 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:00:12.935336 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935094 4926 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:00:12.935336 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935098 4926 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:00:12.935336 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935102 4926 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:00:12.935336 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935106 4926 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:00:12.935336 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935110 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:00:12.935336 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935114 4926 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:00:12.935336 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935117 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:00:12.935336 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935121 4926 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:00:12.935336 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935124 4926 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:00:12.935336 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935128 4926 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:00:12.935336 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935132 4926 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:00:12.935336 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935136 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:00:12.935336 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935139 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:00:12.935336 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935143 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:00:12.935336 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935147 4926 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:00:12.936187 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935151 4926 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:00:12.936187 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935161 4926 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:00:12.936187 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935168 4926 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:00:12.936187 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935172 4926 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:00:12.936187 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935176 4926 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:00:12.936187 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935180 4926 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:00:12.936187 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935184 4926 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:00:12.936187 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935188 4926 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:00:12.936187 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935192 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:00:12.936187 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935196 4926 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:00:12.936187 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935200 4926 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:00:12.936187 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935204 4926 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:00:12.936187 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935209 4926 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:00:12.936187 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935214 4926 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:00:12.936187 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935221 4926 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:00:12.936187 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935227 4926 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:00:12.936187 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935232 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:00:12.936187 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935236 4926 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:00:12.936187 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935240 4926 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:00:12.936998 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935245 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:00:12.936998 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935249 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:00:12.936998 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935253 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:00:12.936998 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935258 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:00:12.936998 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935263 4926 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:00:12.936998 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935267 4926 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:00:12.936998 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935271 4926 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:00:12.936998 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935275 4926 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:00:12.936998 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935279 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:00:12.936998 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935284 4926 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:00:12.936998 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935288 4926 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:00:12.936998 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935292 4926 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:00:12.936998 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935296 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:00:12.936998 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935299 4926 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:00:12.936998 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935306 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:00:12.936998 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935310 4926 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:00:12.936998 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935314 4926 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:00:12.936998 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935318 4926 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:00:12.936998 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935322 4926 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:00:12.937845 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935327 4926 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:00:12.937845 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935331 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:00:12.937845 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935334 4926 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:00:12.937845 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935338 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:00:12.937845 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935342 4926 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:00:12.937845 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935347 4926 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:00:12.937845 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935351 4926 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:00:12.937845 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935355 4926 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:00:12.937845 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935361 4926 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:00:12.937845 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935366 4926 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:00:12.937845 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935372 4926 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:00:12.937845 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935376 4926 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:00:12.937845 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935380 4926 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:00:12.937845 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935384 4926 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:00:12.937845 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935388 4926 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:00:12.937845 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935392 4926 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:00:12.937845 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935397 4926 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:00:12.937845 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935401 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:00:12.937845 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935405 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:00:12.938715 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935409 4926 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:00:12.938715 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935413 4926 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:00:12.938715 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935418 4926 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:00:12.938715 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935422 4926 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:00:12.938715 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935426 4926 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:00:12.938715 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935430 4926 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:00:12.938715 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935434 4926 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:00:12.938715 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935438 4926 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:00:12.938715 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935445 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:00:12.938715 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935449 4926 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:00:12.938715 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935453 4926 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:00:12.938715 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935458 4926 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:00:12.938715 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935462 4926 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:00:12.938715 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935467 4926 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:00:12.938715 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935471 4926 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:00:12.938715 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935475 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:00:12.938715 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935480 4926 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:00:12.938715 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935484 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:00:12.938715 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935488 4926 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:00:12.938715 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935492 4926 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:00:12.939605 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935498 4926 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:00:12.939605 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935502 4926 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:00:12.939605 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935507 4926 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:00:12.939605 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935512 4926 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:00:12.939605 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.935518 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:00:12.939605 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936763 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:00:12.939605 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936775 4926 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:00:12.939605 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936779 4926 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:00:12.939605 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936784 4926 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:00:12.939605 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936788 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:00:12.939605 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936792 4926 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:00:12.939605 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936796 4926 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:00:12.939605 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936801 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:00:12.939605 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936806 4926 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:00:12.939605 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936810 4926 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:00:12.939605 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936814 4926 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:00:12.939605 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936818 4926 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:00:12.939605 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936822 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:00:12.939605 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936826 4926 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:00:12.939605 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936830 4926 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:00:12.940530 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936835 4926 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:00:12.940530 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936840 4926 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:00:12.940530 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936844 4926 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:00:12.940530 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936848 4926 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:00:12.940530 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936852 4926 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:00:12.940530 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936856 4926 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:00:12.940530 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936860 4926 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:00:12.940530 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936864 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:00:12.940530 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936868 4926 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:00:12.940530 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936873 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:00:12.940530 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936876 4926 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:00:12.940530 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936881 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:00:12.940530 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936886 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:00:12.940530 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936890 4926 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:00:12.940530 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936895 4926 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:00:12.940530 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936900 4926 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:00:12.940530 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936904 4926 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:00:12.940530 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936908 4926 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:00:12.940530 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936912 4926 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:00:12.940530 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936917 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:00:12.941408 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936921 4926 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:00:12.941408 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936926 4926 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:00:12.941408 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936930 4926 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:00:12.941408 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936934 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:00:12.941408 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936938 4926 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:00:12.941408 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936943 4926 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:00:12.941408 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936947 4926 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:00:12.941408 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936951 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:00:12.941408 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936955 4926 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:00:12.941408 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936959 4926 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:00:12.941408 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936963 4926 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:00:12.941408 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936968 4926 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:00:12.941408 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936972 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:00:12.941408 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936976 4926 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:00:12.941408 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936980 4926 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:00:12.941408 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936987 4926 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:00:12.941408 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936993 4926 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:00:12.941408 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.936997 4926 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:00:12.941408 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937001 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:00:12.942086 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937005 4926 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:00:12.942086 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937009 4926 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:00:12.942086 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937013 4926 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:00:12.942086 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937018 4926 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:00:12.942086 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937023 4926 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:00:12.942086 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937044 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:00:12.942086 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937050 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:00:12.942086 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937054 4926 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:00:12.942086 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937058 4926 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:00:12.942086 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937063 4926 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:00:12.942086 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937067 4926 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:00:12.942086 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937071 4926 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:00:12.942086 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937075 4926 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:00:12.942086 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937082 4926 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:00:12.942086 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937088 4926 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:00:12.942086 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937093 4926 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:00:12.942086 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937097 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:00:12.942086 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937102 4926 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:00:12.942086 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937106 4926 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:00:12.942086 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937111 4926 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:00:12.942604 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937116 4926 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:00:12.942604 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937120 4926 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:00:12.942604 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937125 4926 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:00:12.942604 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937129 4926 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:00:12.942604 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937133 4926 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:00:12.942604 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937138 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:00:12.942604 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937142 4926 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:00:12.942604 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937146 4926 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:00:12.942604 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937151 4926 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:00:12.942604 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937155 4926 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:00:12.942604 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937159 4926 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:00:12.942604 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937164 4926 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:00:12.942604 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937168 4926 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:00:12.942604 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937172 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:00:12.942604 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937176 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:00:12.942604 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937180 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:00:12.942604 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937185 4926 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:00:12.942604 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937190 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:00:12.942604 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937194 4926 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:00:12.942604 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937198 4926 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:00:12.943091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937202 4926 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:00:12.943091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937206 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:00:12.943091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937211 4926 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:00:12.943091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937215 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:00:12.943091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937219 4926 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:00:12.943091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937224 4926 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:00:12.943091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937229 4926 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:00:12.943091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937234 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:00:12.943091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937237 4926 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:00:12.943091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937241 4926 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:00:12.943091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937246 4926 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:00:12.943091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.937250 4926 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:00:12.943091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938052 4926 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:00:12.943091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938064 4926 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:00:12.943091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938070 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:00:12.943091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938074 4926 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:00:12.943091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938078 4926 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:00:12.943091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938082 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:00:12.943091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938087 4926 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:00:12.943597 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938091 4926 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:00:12.943597 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938095 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:00:12.943597 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938099 4926 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:00:12.943597 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938104 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:00:12.943597 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938111 4926 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:00:12.943597 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938116 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:00:12.943597 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938120 4926 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:00:12.943597 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938124 4926 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:00:12.943597 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938128 4926 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:00:12.943597 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938133 4926 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:00:12.943597 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938137 4926 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:00:12.943597 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938142 4926 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:00:12.943597 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938146 4926 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:00:12.943597 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938150 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:00:12.943597 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938154 4926 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:00:12.943597 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938160 4926 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:00:12.943597 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938164 4926 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:00:12.943597 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938168 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:00:12.943597 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938172 4926 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:00:12.943597 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938176 4926 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:00:12.944103 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938180 4926 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:00:12.944103 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938186 4926 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:00:12.944103 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938190 4926 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:00:12.944103 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938194 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:00:12.944103 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938198 4926 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:00:12.944103 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938202 4926 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:00:12.944103 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938206 4926 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:00:12.944103 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938210 4926 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:00:12.944103 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938215 4926 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:00:12.944103 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938219 4926 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:00:12.944103 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938223 4926 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:00:12.944103 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938228 4926 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:00:12.944103 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938232 4926 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:00:12.944103 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938237 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:00:12.944103 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938241 4926 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:00:12.944103 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938245 4926 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:00:12.944103 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938252 4926 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:00:12.944103 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938256 4926 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:00:12.944103 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938260 4926 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:00:12.944103 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938265 4926 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:00:12.944583 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938269 4926 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:00:12.944583 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938273 4926 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:00:12.944583 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938278 4926 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:00:12.944583 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938282 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:00:12.944583 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938287 4926 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:00:12.944583 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938292 4926 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:00:12.944583 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938296 4926 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:00:12.944583 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938300 4926 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:00:12.944583 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938305 4926 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:00:12.944583 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938309 4926 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:00:12.944583 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938313 4926 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:00:12.944583 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938317 4926 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:00:12.944583 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938321 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:00:12.944583 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938325 4926 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:00:12.944583 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938330 4926 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:00:12.944583 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938334 4926 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:00:12.944583 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938338 4926 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:00:12.944583 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938342 4926 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:00:12.944583 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938346 4926 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:00:12.944583 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938351 4926 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:00:12.945082 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938355 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:00:12.945082 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938359 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:00:12.945082 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938363 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:00:12.945082 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938368 4926 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:00:12.945082 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938372 4926 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:00:12.945082 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938376 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:00:12.945082 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938381 4926 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:00:12.945082 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938384 4926 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:00:12.945082 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938391 4926 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:00:12.945082 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938395 4926 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:00:12.945082 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938402 4926 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:00:12.945082 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938408 4926 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:00:12.945082 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938412 4926 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:00:12.945082 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938416 4926 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:00:12.945082 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938421 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:00:12.945082 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938425 4926 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:00:12.945082 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938429 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:00:12.945082 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938433 4926 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:00:12.945508 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938438 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:00:12.945508 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938442 4926 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:00:12.945508 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938446 4926 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:00:12.945508 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938450 4926 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:00:12.945508 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938454 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:00:12.945508 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938458 4926 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:00:12.945508 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938463 4926 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:00:12.945508 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938468 4926 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:00:12.945508 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938474 4926 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:00:12.945508 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938478 4926 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:00:12.945508 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938484 4926 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:00:12.945508 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938488 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:00:12.945508 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938492 4926 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:00:12.945508 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938496 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:00:12.945508 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938500 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:00:12.945508 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938504 4926 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:00:12.945508 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938508 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:00:12.945508 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938513 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:00:12.945508 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938517 4926 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938521 4926 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.938526 4926 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938661 4926 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938674 4926 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938684 4926 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938690 4926 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938697 4926 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938702 4926 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938709 4926 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938714 4926 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938719 4926 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938724 4926 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938729 4926 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938734 4926 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938738 4926 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938743 4926 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938747 4926 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938753 4926 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938758 4926 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938771 4926 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938779 4926 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938783 4926 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938788 4926 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:00:12.945956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938793 4926 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938798 4926 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938805 4926 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938810 4926 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938815 4926 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938820 4926 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938825 4926 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938832 4926 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938838 4926 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938843 4926 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938848 4926 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938855 4926 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938861 4926 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938866 4926 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938870 4926 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938875 4926 flags.go:64] FLAG: --enable-server="true" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938880 4926 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938887 4926 flags.go:64] FLAG: --event-burst="100" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938891 4926 flags.go:64] FLAG: --event-qps="50" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938896 4926 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938901 4926 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938905 4926 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938912 4926 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938917 4926 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938922 4926 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:00:12.946538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938926 4926 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938931 4926 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938937 4926 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938942 4926 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938946 4926 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938951 4926 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938955 4926 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938959 4926 flags.go:64] FLAG: --feature-gates="" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938970 4926 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938975 4926 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938980 4926 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938985 4926 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938990 4926 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.938995 4926 flags.go:64] FLAG: --help="false" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939002 4926 flags.go:64] FLAG: --hostname-override="ip-10-0-142-3.ec2.internal" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939007 4926 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939012 4926 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939017 4926 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939023 4926 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939045 4926 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939051 4926 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939055 4926 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939060 4926 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:00:12.947182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939065 4926 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939069 4926 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939075 4926 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939079 4926 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939084 4926 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939088 4926 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939093 4926 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939098 4926 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939102 4926 flags.go:64] FLAG: --lock-file="" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939106 4926 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939111 4926 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939116 4926 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939126 4926 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939131 4926 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939135 4926 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939140 4926 flags.go:64] FLAG: --logging-format="text" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939144 4926 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939149 4926 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939154 4926 flags.go:64] FLAG: --manifest-url="" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939158 4926 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939165 4926 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939170 4926 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939176 4926 flags.go:64] FLAG: --max-pods="110" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939184 4926 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939189 4926 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:00:12.947726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939194 4926 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939198 4926 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939204 4926 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939211 4926 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939216 4926 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939227 4926 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939232 4926 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939236 4926 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939241 4926 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939246 4926 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939251 4926 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939255 4926 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939260 4926 flags.go:64] FLAG: --port="10250" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939265 4926 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939269 4926 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0953d1f9b975c2ff4" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939274 4926 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939279 4926 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939284 4926 flags.go:64] FLAG: --register-node="true" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939289 4926 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939295 4926 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939300 4926 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939305 4926 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939310 4926 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939316 4926 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939320 4926 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939325 4926 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:00:12.948340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939330 4926 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939334 4926 flags.go:64] FLAG: --runonce="false" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939339 4926 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939344 4926 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939350 4926 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939356 4926 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939361 4926 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939366 4926 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939370 4926 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939377 4926 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939382 4926 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939386 4926 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939391 4926 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939396 4926 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939400 4926 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939405 4926 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939410 4926 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939418 4926 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939423 4926 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939427 4926 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939434 4926 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939439 4926 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939443 4926 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939448 4926 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939453 4926 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:00:12.948953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939457 4926 flags.go:64] FLAG: --v="2" Oct 30 21:00:12.949588 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939464 4926 flags.go:64] FLAG: --version="false" Oct 30 21:00:12.949588 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939471 4926 flags.go:64] FLAG: --vmodule="" Oct 30 21:00:12.949588 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939477 4926 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:00:12.949588 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.939482 4926 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:00:12.949588 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940024 4926 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:00:12.949588 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940054 4926 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:00:12.949588 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940061 4926 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:00:12.949588 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940066 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:00:12.949588 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940071 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:00:12.949588 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940076 4926 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:00:12.949588 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940083 4926 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:00:12.949588 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940088 4926 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:00:12.949588 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940093 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:00:12.949588 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940104 4926 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:00:12.949588 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940109 4926 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:00:12.949588 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940116 4926 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:00:12.949588 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940120 4926 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:00:12.949588 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940125 4926 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:00:12.949588 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940129 4926 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:00:12.949588 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940133 4926 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:00:12.950172 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940138 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:00:12.950172 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940142 4926 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:00:12.950172 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940147 4926 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:00:12.950172 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940151 4926 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:00:12.950172 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940156 4926 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:00:12.950172 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940160 4926 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:00:12.950172 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940169 4926 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:00:12.950172 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940173 4926 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:00:12.950172 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940177 4926 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:00:12.950172 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940182 4926 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:00:12.950172 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940186 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:00:12.950172 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940191 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:00:12.950172 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940195 4926 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:00:12.950172 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940200 4926 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:00:12.950172 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940206 4926 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:00:12.950172 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940210 4926 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:00:12.950172 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940214 4926 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:00:12.950172 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940219 4926 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:00:12.950172 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940230 4926 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:00:12.950172 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940235 4926 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:00:12.950697 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940240 4926 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:00:12.950697 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940244 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:00:12.950697 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940251 4926 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:00:12.950697 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940255 4926 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:00:12.950697 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940259 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:00:12.950697 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940264 4926 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:00:12.950697 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940268 4926 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:00:12.950697 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940275 4926 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:00:12.950697 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940280 4926 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:00:12.950697 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940285 4926 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:00:12.950697 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940289 4926 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:00:12.950697 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940299 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:00:12.950697 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940303 4926 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:00:12.950697 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940307 4926 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:00:12.950697 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940312 4926 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:00:12.950697 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940316 4926 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:00:12.950697 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940320 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:00:12.950697 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940325 4926 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:00:12.950697 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940332 4926 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:00:12.951182 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940338 4926 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:00:12.951182 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940343 4926 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:00:12.951182 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940348 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:00:12.951182 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940353 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:00:12.951182 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940362 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:00:12.951182 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940367 4926 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:00:12.951182 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940371 4926 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:00:12.951182 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940378 4926 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:00:12.951182 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940386 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:00:12.951182 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940391 4926 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:00:12.951182 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940395 4926 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:00:12.951182 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940399 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:00:12.951182 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940404 4926 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:00:12.951182 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940409 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:00:12.951182 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940414 4926 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:00:12.951182 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940421 4926 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:00:12.951182 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940425 4926 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:00:12.951182 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940435 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:00:12.951182 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940440 4926 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:00:12.951637 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940444 4926 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:00:12.951637 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940452 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:00:12.951637 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940457 4926 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:00:12.951637 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940462 4926 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:00:12.951637 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940466 4926 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:00:12.951637 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940471 4926 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:00:12.951637 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940476 4926 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:00:12.951637 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940480 4926 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:00:12.951637 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940485 4926 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:00:12.951637 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940489 4926 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:00:12.951637 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940498 4926 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:00:12.951637 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940503 4926 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:00:12.951637 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940507 4926 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:00:12.951637 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940514 4926 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:00:12.951637 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940519 4926 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:00:12.951637 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940523 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:00:12.951637 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940549 4926 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:00:12.951637 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940554 4926 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:00:12.951637 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940558 4926 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:00:12.951637 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940563 4926 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:00:12.952243 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940578 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:00:12.952243 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940583 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:00:12.952243 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940687 4926 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:00:12.952243 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940693 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:00:12.952243 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940698 4926 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:00:12.952243 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940702 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:00:12.952243 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940707 4926 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:00:12.952243 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940712 4926 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:00:12.952243 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940718 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:00:12.952243 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940722 4926 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:00:12.952243 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940727 4926 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:00:12.952243 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940731 4926 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:00:12.952243 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.940778 4926 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:00:12.952243 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940839 4926 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:00:12.952243 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940849 4926 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:00:12.952243 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940854 4926 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:00:12.952688 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940859 4926 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:00:12.952688 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940865 4926 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:00:12.952688 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940869 4926 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:00:12.952688 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940874 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:00:12.952688 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940879 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:00:12.952688 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940883 4926 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:00:12.952688 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940888 4926 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:00:12.952688 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940893 4926 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:00:12.952688 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940897 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:00:12.952688 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940901 4926 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:00:12.952688 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940906 4926 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:00:12.952688 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940910 4926 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:00:12.952688 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940914 4926 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:00:12.952688 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940918 4926 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:00:12.952688 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940922 4926 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:00:12.952688 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940926 4926 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:00:12.952688 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940930 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:00:12.952688 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940934 4926 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:00:12.952688 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940939 4926 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:00:12.952688 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940943 4926 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:00:12.953244 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940947 4926 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:00:12.953244 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940951 4926 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:00:12.953244 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940955 4926 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:00:12.953244 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940960 4926 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:00:12.953244 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940964 4926 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:00:12.953244 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940968 4926 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:00:12.953244 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940972 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:00:12.953244 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940977 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:00:12.953244 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940981 4926 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:00:12.953244 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940985 4926 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:00:12.953244 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940989 4926 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:00:12.953244 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940993 4926 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:00:12.953244 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.940997 4926 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:00:12.953244 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941002 4926 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:00:12.953244 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941006 4926 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:00:12.953244 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941009 4926 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:00:12.953244 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941014 4926 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:00:12.953244 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941018 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:00:12.953244 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941023 4926 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:00:12.953244 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941052 4926 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:00:12.953729 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941057 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:00:12.953729 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941061 4926 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:00:12.953729 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941065 4926 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:00:12.953729 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941069 4926 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:00:12.953729 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941073 4926 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:00:12.953729 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941077 4926 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:00:12.953729 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941081 4926 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:00:12.953729 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941086 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:00:12.953729 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941090 4926 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:00:12.953729 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941094 4926 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:00:12.953729 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941098 4926 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:00:12.953729 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941102 4926 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:00:12.953729 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941106 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:00:12.953729 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941110 4926 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:00:12.953729 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941119 4926 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:00:12.953729 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941124 4926 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:00:12.953729 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941129 4926 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:00:12.953729 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941133 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:00:12.953729 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941137 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:00:12.954183 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941142 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:00:12.954183 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941146 4926 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:00:12.954183 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941150 4926 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:00:12.954183 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941156 4926 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:00:12.954183 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941161 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:00:12.954183 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941166 4926 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:00:12.954183 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941170 4926 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:00:12.954183 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941174 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:00:12.954183 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941178 4926 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:00:12.954183 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941183 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:00:12.954183 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941187 4926 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:00:12.954183 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941192 4926 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:00:12.954183 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941196 4926 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:00:12.954183 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941201 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:00:12.954183 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941206 4926 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:00:12.954183 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941210 4926 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:00:12.954183 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941214 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:00:12.954183 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941218 4926 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:00:12.954183 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941222 4926 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:00:12.954728 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941227 4926 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:00:12.954728 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941231 4926 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:00:12.954728 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941236 4926 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:00:12.954728 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941240 4926 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:00:12.954728 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941244 4926 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:00:12.954728 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941248 4926 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:00:12.954728 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941252 4926 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:00:12.954728 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941257 4926 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:00:12.954728 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941261 4926 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:00:12.954728 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941265 4926 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:00:12.954728 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941269 4926 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:00:12.954728 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941273 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:00:12.954728 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941277 4926 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:00:12.954728 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941282 4926 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:00:12.954728 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941286 4926 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:00:12.954728 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941290 4926 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:00:12.954728 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941294 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:00:12.954728 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941298 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:00:12.954728 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941303 4926 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:00:12.954728 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941307 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:00:12.955471 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941311 4926 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:00:12.955471 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941318 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:00:12.955471 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941322 4926 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:00:12.955471 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941327 4926 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:00:12.955471 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.941333 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:00:12.955471 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.945086 4926 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:00:12.955471 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.945103 4926 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:00:12.955471 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.945121 4926 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:00:12.955471 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.945126 4926 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:00:12.955471 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945180 4926 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:00:12.955471 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945185 4926 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:00:12.955471 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945188 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:00:12.955471 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945191 4926 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:00:12.955471 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945195 4926 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:00:12.955471 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945197 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:00:12.955471 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945200 4926 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:00:12.955471 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945203 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:00:12.955471 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945206 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:00:12.955471 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945209 4926 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:00:12.955471 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945211 4926 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:00:12.956249 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945214 4926 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:00:12.956249 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945217 4926 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:00:12.956249 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945219 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:00:12.956249 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945222 4926 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:00:12.956249 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945226 4926 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:00:12.956249 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945230 4926 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:00:12.956249 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945233 4926 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:00:12.956249 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945236 4926 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:00:12.956249 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945238 4926 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:00:12.956249 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945241 4926 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:00:12.956249 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945243 4926 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:00:12.956249 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945246 4926 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:00:12.956249 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945250 4926 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:00:12.956249 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945252 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:00:12.956249 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945255 4926 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:00:12.956249 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945258 4926 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:00:12.956249 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945261 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:00:12.956249 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945263 4926 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:00:12.956249 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945267 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:00:12.956777 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945270 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:00:12.956777 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945273 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:00:12.956777 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945276 4926 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:00:12.956777 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945278 4926 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:00:12.956777 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945281 4926 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:00:12.956777 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945283 4926 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:00:12.956777 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945286 4926 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:00:12.956777 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945288 4926 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:00:12.956777 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945291 4926 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:00:12.956777 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945293 4926 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:00:12.956777 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945296 4926 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:00:12.956777 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945298 4926 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:00:12.956777 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945301 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:00:12.956777 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945303 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:00:12.956777 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945306 4926 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:00:12.956777 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945308 4926 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:00:12.956777 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945311 4926 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:00:12.956777 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945313 4926 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:00:12.956777 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945316 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:00:12.956777 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945319 4926 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:00:12.957407 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945321 4926 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:00:12.957407 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945324 4926 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:00:12.957407 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945327 4926 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:00:12.957407 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945330 4926 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:00:12.957407 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945332 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:00:12.957407 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945335 4926 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:00:12.957407 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945337 4926 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:00:12.957407 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945340 4926 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:00:12.957407 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945343 4926 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:00:12.957407 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945346 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:00:12.957407 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945349 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:00:12.957407 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945351 4926 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:00:12.957407 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945354 4926 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:00:12.957407 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945357 4926 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:00:12.957407 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945360 4926 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:00:12.957407 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945362 4926 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:00:12.957407 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945364 4926 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:00:12.957407 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945367 4926 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:00:12.957407 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945369 4926 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:00:12.957407 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945372 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:00:12.957872 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945374 4926 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:00:12.957872 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945377 4926 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:00:12.957872 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945379 4926 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:00:12.957872 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945382 4926 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:00:12.957872 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945385 4926 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:00:12.957872 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945387 4926 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:00:12.957872 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945389 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:00:12.957872 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945392 4926 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:00:12.957872 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945395 4926 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:00:12.957872 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945397 4926 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:00:12.957872 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945400 4926 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:00:12.957872 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945403 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:00:12.957872 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945406 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:00:12.957872 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945409 4926 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:00:12.957872 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945413 4926 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:00:12.957872 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945416 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:00:12.957872 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945419 4926 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:00:12.957872 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945422 4926 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:00:12.957872 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945425 4926 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:00:12.958340 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945427 4926 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:00:12.958340 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945430 4926 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:00:12.958340 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945433 4926 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:00:12.958340 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945435 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:00:12.958340 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945438 4926 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:00:12.958340 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945440 4926 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:00:12.958340 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945443 4926 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:00:12.958340 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945446 4926 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:00:12.958340 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945449 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:00:12.958340 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945451 4926 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:00:12.958340 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945453 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:00:12.958340 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945456 4926 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:00:12.958340 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945458 4926 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:00:12.958340 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945461 4926 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:00:12.958340 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945463 4926 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:00:12.958340 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945466 4926 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:00:12.958340 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945468 4926 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:00:12.958735 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.945495 4926 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:00:12.958735 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945638 4926 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:00:12.958735 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945642 4926 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:00:12.958735 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945645 4926 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:00:12.958735 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945648 4926 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:00:12.958735 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945650 4926 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:00:12.958735 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945653 4926 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:00:12.958735 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945655 4926 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:00:12.958735 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945658 4926 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:00:12.958735 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945660 4926 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:00:12.958735 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945663 4926 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:00:12.958735 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945666 4926 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:00:12.958735 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945668 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:00:12.958735 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945672 4926 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:00:12.958735 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945676 4926 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:00:12.958735 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945679 4926 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:00:12.959159 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945681 4926 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:00:12.959159 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945684 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:00:12.959159 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945687 4926 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:00:12.959159 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945689 4926 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:00:12.959159 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945692 4926 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:00:12.959159 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945695 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:00:12.959159 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945698 4926 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:00:12.959159 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945701 4926 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:00:12.959159 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945703 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:00:12.959159 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945706 4926 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:00:12.959159 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945708 4926 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:00:12.959159 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945711 4926 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:00:12.959159 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945714 4926 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:00:12.959159 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945716 4926 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:00:12.959159 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945719 4926 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:00:12.959159 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945721 4926 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:00:12.959159 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945723 4926 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:00:12.959159 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945726 4926 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:00:12.959159 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945728 4926 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:00:12.959159 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945731 4926 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:00:12.959615 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945733 4926 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:00:12.959615 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945736 4926 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:00:12.959615 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945738 4926 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:00:12.959615 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945741 4926 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:00:12.959615 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945743 4926 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:00:12.959615 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945746 4926 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:00:12.959615 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945748 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:00:12.959615 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945751 4926 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:00:12.959615 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945754 4926 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:00:12.959615 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945756 4926 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:00:12.959615 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945759 4926 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:00:12.959615 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945761 4926 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:00:12.959615 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945763 4926 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:00:12.959615 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945766 4926 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:00:12.959615 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945768 4926 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:00:12.959615 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945771 4926 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:00:12.959615 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945773 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:00:12.959615 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945776 4926 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:00:12.959615 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945778 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:00:12.959615 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945781 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:00:12.960091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945784 4926 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:00:12.960091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945786 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:00:12.960091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945788 4926 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:00:12.960091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945791 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:00:12.960091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945793 4926 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:00:12.960091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945796 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:00:12.960091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945799 4926 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:00:12.960091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945801 4926 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:00:12.960091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945803 4926 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:00:12.960091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945806 4926 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:00:12.960091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945808 4926 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:00:12.960091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945810 4926 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:00:12.960091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945813 4926 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:00:12.960091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945815 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:00:12.960091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945818 4926 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:00:12.960091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945820 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:00:12.960091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945823 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:00:12.960091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945825 4926 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:00:12.960091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945828 4926 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:00:12.960091 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945831 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:00:12.960568 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945833 4926 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:00:12.960568 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945836 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:00:12.960568 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945839 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:00:12.960568 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945841 4926 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:00:12.960568 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945843 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:00:12.960568 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945846 4926 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:00:12.960568 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945849 4926 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:00:12.960568 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945851 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:00:12.960568 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945854 4926 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:00:12.960568 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945856 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:00:12.960568 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945859 4926 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:00:12.960568 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945861 4926 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:00:12.960568 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945864 4926 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:00:12.960568 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945866 4926 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:00:12.960568 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945869 4926 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:00:12.960568 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945874 4926 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:00:12.960568 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945877 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:00:12.960568 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945880 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:00:12.960568 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945882 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:00:12.961097 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945885 4926 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:00:12.961097 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945887 4926 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:00:12.961097 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945890 4926 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:00:12.961097 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945892 4926 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:00:12.961097 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945895 4926 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:00:12.961097 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945897 4926 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:00:12.961097 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945900 4926 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:00:12.961097 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945905 4926 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:00:12.961097 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945909 4926 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:00:12.961097 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945912 4926 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:00:12.961097 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945915 4926 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:00:12.961097 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945918 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:00:12.961097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.945940 4926 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:00:12.961097 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945970 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:00:12.961097 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945974 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:00:12.961480 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945976 4926 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:00:12.961480 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945979 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:00:12.961480 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945982 4926 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:00:12.961480 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945984 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:00:12.961480 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945987 4926 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:00:12.961480 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945990 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:00:12.961480 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945992 4926 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:00:12.961480 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945995 4926 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:00:12.961480 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.945997 4926 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:00:12.961480 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946000 4926 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:00:12.961480 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946002 4926 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:00:12.961480 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946006 4926 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:00:12.961480 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946008 4926 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:00:12.961480 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946011 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:00:12.961480 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946015 4926 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:00:12.961480 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946018 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:00:12.961480 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946020 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:00:12.961480 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946023 4926 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:00:12.961480 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946025 4926 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:00:12.961480 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946028 4926 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:00:12.961949 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946045 4926 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:00:12.961949 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946049 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:00:12.961949 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946053 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:00:12.961949 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946057 4926 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:00:12.961949 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946062 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:00:12.961949 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946065 4926 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:00:12.961949 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946068 4926 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:00:12.961949 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946070 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:00:12.961949 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946073 4926 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:00:12.961949 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946075 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:00:12.961949 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946078 4926 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:00:12.961949 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946081 4926 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:00:12.961949 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946083 4926 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:00:12.961949 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946085 4926 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:00:12.961949 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946088 4926 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:00:12.961949 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946090 4926 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:00:12.961949 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946093 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:00:12.961949 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946095 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:00:12.961949 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946098 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:00:12.962415 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946101 4926 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:00:12.962415 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946103 4926 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:00:12.962415 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946106 4926 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:00:12.962415 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946108 4926 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:00:12.962415 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946111 4926 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:00:12.962415 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946114 4926 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:00:12.962415 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946117 4926 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:00:12.962415 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946122 4926 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:00:12.962415 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946125 4926 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:00:12.962415 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946128 4926 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:00:12.962415 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946131 4926 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:00:12.962415 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946133 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:00:12.962415 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946136 4926 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:00:12.962415 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946138 4926 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:00:12.962415 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946141 4926 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:00:12.962415 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946144 4926 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:00:12.962415 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946146 4926 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:00:12.962415 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946150 4926 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:00:12.962415 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946152 4926 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:00:12.962897 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946155 4926 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:00:12.962897 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946159 4926 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:00:12.962897 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946161 4926 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:00:12.962897 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946164 4926 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:00:12.962897 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946166 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:00:12.962897 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946170 4926 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:00:12.962897 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946173 4926 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:00:12.962897 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946176 4926 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:00:12.962897 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946180 4926 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:00:12.962897 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946183 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:00:12.962897 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946186 4926 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:00:12.962897 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946189 4926 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:00:12.962897 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946192 4926 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:00:12.962897 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946195 4926 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:00:12.962897 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946198 4926 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:00:12.962897 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946200 4926 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:00:12.962897 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946203 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:00:12.962897 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946205 4926 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:00:12.962897 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946208 4926 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:00:12.963362 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946211 4926 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:00:12.963362 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946215 4926 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:00:12.963362 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946218 4926 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:00:12.963362 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946221 4926 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:00:12.963362 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946223 4926 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:00:12.963362 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946226 4926 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:00:12.963362 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946229 4926 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:00:12.963362 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946231 4926 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:00:12.963362 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946234 4926 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:00:12.963362 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946236 4926 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:00:12.963362 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946240 4926 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:00:12.963362 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946245 4926 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:00:12.963362 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946247 4926 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:00:12.963362 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946250 4926 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:00:12.963362 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946252 4926 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:00:12.963362 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946255 4926 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:00:12.963362 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946257 4926 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:00:12.963362 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946260 4926 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:00:12.963362 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946262 4926 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:00:12.963362 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946265 4926 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:00:12.963835 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946267 4926 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:00:12.963835 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946270 4926 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:00:12.963835 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946272 4926 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:00:12.963835 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946274 4926 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:00:12.963835 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946277 4926 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:00:12.963835 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946279 4926 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:00:12.963835 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:12.946282 4926 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:00:12.963835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.946338 4926 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:00:12.963835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.947326 4926 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:00:12.963835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.947393 4926 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:00:12.963835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.947569 4926 server.go:1014] "Starting client certificate rotation" Oct 30 21:00:12.963835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.947705 4926 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:00:12.963835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.947756 4926 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 20:54:32 +0000 UTC" deadline="2028-07-29 11:04:21.180753744 +0000 UTC" Oct 30 21:00:12.963835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.947787 4926 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15302h4m8.232969746s" Oct 30 21:00:12.963835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.948019 4926 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:00:12.963835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.949060 4926 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:00:12.964258 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.950183 4926 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:00:12.964258 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.951735 4926 log.go:25] "Validated CRI v1 image API" Oct 30 21:00:12.964258 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.952477 4926 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:00:12.964258 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.954706 4926 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a2e7b3ab-3d6d-4b22-9176-ef965c843830:/dev/nvme0n1p3 b79046f5-774a-4003-b682-b478f32876d5:/dev/nvme0n1p4] Oct 30 21:00:12.964386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.954726 4926 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03dab2bd4f5ccd547d64bb7a5331444dfb48a537d0ebfa55136fc411976c629d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03dab2bd4f5ccd547d64bb7a5331444dfb48a537d0ebfa55136fc411976c629d/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1ba5a3b6a76258cbeaf8a5c8af034efbdeb40e73ef0fe0493356c3d6b056133f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1ba5a3b6a76258cbeaf8a5c8af034efbdeb40e73ef0fe0493356c3d6b056133f/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4307f887bef1e1ad6367874facafc8cffd4c890d6415ff5c3dd10da77ae5b8f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4307f887bef1e1ad6367874facafc8cffd4c890d6415ff5c3dd10da77ae5b8f9/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4961a255689d4f8b3ca3f78ac06b9076bfbd9659ab98826173df297fdb40eda9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4961a255689d4f8b3ca3f78ac06b9076bfbd9659ab98826173df297fdb40eda9/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5ae83504a960547e4f577294464b10264d64c8b625de5f99452e0f3ccda56293/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5ae83504a960547e4f577294464b10264d64c8b625de5f99452e0f3ccda56293/userdata/shm major:0 minor:334 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/635e766e79e31aed78f48ee33e9c38219337c8af31bced336183205b22c540ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/635e766e79e31aed78f48ee33e9c38219337c8af31bced336183205b22c540ad/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/81d54f22853849278d8b2c649c0d4a515ab479ecfeef0013099e3f3f5dc690c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/81d54f22853849278d8b2c649c0d4a515ab479ecfeef0013099e3f3f5dc690c5/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ae91fed0dc7feaa07ff952a02edbe8cd026ec5af13f2e6e31a2a30d5a9015e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ae91fed0dc7feaa07ff952a02edbe8cd026ec5af13f2e6e31a2a30d5a9015e7/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/93d288e4555cd3e2cc0345c8afcd8156f019e9f1443044bb18e3060667bd1079/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/93d288e4555cd3e2cc0345c8afcd8156f019e9f1443044bb18e3060667bd1079/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d5720d79d5bc05f502219a5b16458606d49b67752f9379196b789c214d91311/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d5720d79d5bc05f502219a5b16458606d49b67752f9379196b789c214d91311/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9fd2d7bbece44d55d7b897ad7783d27cac24ed24ae77eeed57b1a3de942d3cb5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9fd2d7bbece44d55d7b897ad7783d27cac24ed24ae77eeed57b1a3de942d3cb5/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae4fa48a1d708a46447ff5ac1999bb738e3759a7b20711ec7e5b1472cc0a5b61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae4fa48a1d708a46447ff5ac1999bb738e3759a7b20711ec7e5b1472cc0a5b61/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf089dbea2ba22737b1bb71e42206d667d7b745dcaa421f6d6abca101caa3ceb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf089dbea2ba22737b1bb71e42206d667d7b745dcaa421f6d6abca101caa3ceb/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c67809ee48958159646ad059a290b0559a8e206498105424fc67d98b5537cd83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c67809ee48958159646ad059a290b0559a8e206498105424fc67d98b5537cd83/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c9e921e24c64a15435106455241e0ff8bfd38a668102ecd4b56609cc4ed29f10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c9e921e24c64a15435106455241e0ff8bfd38a668102ecd4b56609cc4ed29f10/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d122169f6e92689276586e461c8d1c60c93d73f4f62ef18a38b79104e603723b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d122169f6e92689276586e461c8d1c60c93d73f4f62ef18a38b79104e603723b/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d9daf948fdbabb2d3d1e892ca5c110c1242fbad1590c0fd045f06fb61391b932/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d9daf948fdbabb2d3d1e892ca5c110c1242fbad1590c0fd045f06fb61391b932/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e19a827d04e419c8553f530ce0591dd5284d48acb9b1ae403d0ced5d7909c7ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e19a827d04e419c8553f530ce0591dd5284d48acb9b1ae403d0ced5d7909c7ba/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e1e8b14579a11d4ed7ed03b3a11a0ce6ebdd17b3bc316392fe600bb918be48c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e1e8b14579a11d4ed7ed03b3a11a0ce6ebdd17b3bc316392fe600bb918be48c9/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e63373561db72954e6ed55d6d4a214d939a57c30e5f981211acb86b0b653ab7f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e63373561db72954e6ed55d6d4a214d939a57c30e5f981211acb86b0b653ab7f/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7dd4d31803010cf19be81325287be8a522d556865e760858b16bfe0f4439421/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7dd4d31803010cf19be81325287be8a522d556865e760858b16bfe0f4439421/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fcc5c3c5eb56c860c84b8d31c502139f42a747a5ad1b6e0105faf9366a948c44/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fcc5c3c5eb56c860c84b8d31c502139f42a747a5ad1b6e0105faf9366a948c44/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fdbd72ddb43ee4578e925ba88e43c3fb79e43f8fa5709ce70d51039ec2502586/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fdbd72ddb43ee4578e925ba88e43c3fb79e43f8fa5709ce70d51039ec2502586/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03e64eb3-e7df-464d-9382-47cd0e570456/volumes/kubernetes.io~projected/kube-api-access-9xcm4:{mountpoint:/var/lib/kubelet/pods/03e64eb3-e7df-464d-9382-47cd0e570456/volumes/kubernetes.io~projected/kube-api-access-9xcm4 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b77d4c7-37eb-4bad-8367-31346289726e/volumes/kubernetes.io~projected/kube-api-access-l2749:{mountpoint:/var/lib/kubelet/pods/0b77d4c7-37eb-4bad-8367-31346289726e/volumes/kubernetes.io~projected/kube-api-access-l2749 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0f0b2878-417e-4986-8067-b50a16d16967/volumes/kubernetes.io~projected/kube-api-access-z2k65:{mountpoint:/var/lib/kubelet/pods/0f0b2878-417e-4986-8067-b50a16d16967/volumes/kubernetes.io~projected/kube-api-access-z2k65 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1362c712-7332-4a6e-a215-633d2f02acca/volumes/kubernetes.io~projected/kube-api-access-dhhrx:{mountpoint:/var/lib/kubelet/pods/1362c712-7332-4a6e-a215-633d2f02acca/volumes/kubernetes.io~projected/kube-api-access-dhhrx major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a2fb9ac-a905-4e20-b1be-7928d5e7881a/volumes/kubernetes.io~projected/kube-api-access-pwf67:{mountpoint:/var/lib/kubelet/pods/1a2fb9ac-a905-4e20-b1be-7928d5e7881a/volumes/kubernetes.io~projected/kube-api-access-pwf67 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a2fb9ac-a905-4e20-b1be-7928d5e7881a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/1a2fb9ac-a905-4e20-b1be-7928d5e7881a/volumes/kubernetes.io~secret/serving-cert major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e20348d-9970-4de7-80be-84f29ef8ece6/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/1e20348d-9970-4de7-80be-84f29ef8ece6/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e20348d-9970-4de7-80be-84f29ef8ece6/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/1e20348d-9970-4de7-80be-84f29ef8ece6/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e20348d-9970-4de7-80be-84f29ef8ece6/volumes/kubernetes.io~projected/kube-api-access-d58ss:{mountpoint:/var/lib/kubelet/pods/1e20348d-9970-4de7-80be-84f29ef8ece6/volumes/kubernetes.io~projected/kube-api-access-d58ss major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d/volumes/kubernetes.io~projected/kube-api-access-cz752:{mountpoint:/var/lib/kubelet/pods/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d/volumes/kubernetes.io~projected/kube-api-access-cz752 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d/volumes/kubernetes.io~secret/default-certificate major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d/volumes/kubernetes.io~secret/stats-auth major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3358db14-a01d-479e-a742-bdcd1ca80527/volumes/kubernetes.io~projected/kube-api-access-cgg8n:{mountpoint:/var/lib/kubelet/pods/3358db14-a01d-479e-a742-bdcd1ca80527/volumes/kubernetes.io~projected/kube-api-access-cgg8n major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4901add7-3312-4950-ae4c-4d6581380e9e/volumes/kubernetes.io~projected/kube-api-access-4w859:{mountpoint:/var/lib/kubelet/pods/4901add7-3312-4950-ae4c-4d6581380e9e/volumes/kubernetes.io~projected/kube-api-access-4w859 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4901add7-3312-4950-ae4c-4d6581380e9e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/4901add7-3312-4950-ae4c-4d6581380e9e/volumes/kubernetes.io~secret/cert major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ce43826-7550-4c62-8acc-48714541e426/volumes/kubernetes.io~projected/kube-api-access-gvl2l:{mountpoint:/var/lib/kubelet/pods/4ce43826-7550-4c62-8acc-48714541e426/volumes/kubernetes.io~projected/kube-api-access-gvl2l major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4ce43826-7550-4c62-8acc-48714541e426/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/4ce43826-7550-4c62-8acc-48714541e426/volumes/kubernetes.io~secret/serving-cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5/volumes/kubernetes.io~projected/kube-api-access-dz6tt:{mountpoint:/var/lib/kubelet/pods/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5/volumes/kubernetes.io~projected/kube-api-access-dz6tt major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/662ead00-b524-4a6e-815b-c3399b3c9e6c/volumes/kubernetes.io~projected/kube-api-access-6tf9r:{mountpoint:/var/lib/kubelet/pods/662ead00-b524-4a6e-815b-c3399b3c9e6c/volumes/kubernetes.io~projected/kube-api-access-6tf9r major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b4ba52c-ad04-4a25-9158-89941367abd3/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/6b4ba52c-ad04-4a25-9158-89941367abd3/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b4ba52c-ad04-4a25-9158-89941367abd3/volumes/kubernetes.io~projected/kube-api-access-vrt6b:{mountpoint:/var/lib/kubelet/pods/6b4ba52c-ad04-4a25-9158-89941367abd3/volumes/kubernetes.io~projected/kube-api-access-vrt6b major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b4ba52c-ad04-4a25-9158-89941367abd3/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/6b4ba52c-ad04-4a25-9158-89941367abd3/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6b4ba52c-ad04-4a25-9158-89941367abd3/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/6b4ba52c-ad04-4a25-9158-89941367abd3/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d9c7935-1d3d-4323-ac72-008a394e7f31/volumes/kubernetes.io~projected/kube-api-access-nmcw2:{mountpoint:/var/lib/kubelet/pods/6d9c7935-1d3d-4323-ac72-008a394e7f31/volumes/kubernetes.io~projected/kube-api-access-nmcw2 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/705e2b0e-10e1-4710-8dfb-bbd862c8c084/volumes/kubernetes.io~projected/kube-api-access-x6gqs:{mountpoint:/var/lib/kubelet/pods/705e2b0e-10e1-4710-8dfb-bbd862c8c084/volumes/kubernetes.io~projected/kube-api-access-x6gqs major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/705e2b0e-10e1-4710-8dfb-bbd862c8c084/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/705e2b0e-10e1-4710-8dfb-bbd862c8c084/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88f212cc-4eb8-4a49-9c13-a1d395ab43c6/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/88f212cc-4eb8-4a49-9c13-a1d395ab43c6/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88f212cc-4eb8-4a49-9c13-a1d395ab43c6/volumes/kubernetes.io~projected/kube-api-access-67blk:{mountpoint:/var/lib/kubelet/pods/88f212cc-4eb8-4a49-9c13-a1d395ab43c6/volumes/kubernetes.io~projected/kube-api-access-67blk major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88f212cc-4eb8-4a49-9c13-a1d395ab43c6/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/88f212cc-4eb8-4a49-9c13-a1d395ab43c6/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8bc258c0-5d93-4ddb-b49c-9a06167f6185/volumes/kubernetes.io~projected/kube-api-access-x799c:{mountpoint:/var/lib/kubelet/pods/8bc258c0-5d93-4ddb-b49c-9a06167f6185/volumes/kubernetes.io~projected/kube-api-access-x799c major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93f2c084-a1d2-4bda-8a41-c01829cc81ba/volumes/kubernetes.io~projected/kube-api-access-6g67v:{mountpoint:/var/lib/kubelet/pods/93f2c084-a1d2-4bda-8a41-c01829cc81ba/volumes/kubernetes.io~projected/kube-api-access-6g67v major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97872ff8-a1f5-4062-8202-41aa9d8a93bc/volumes/kubernetes.io~projected/kube-api-access-k2tfx:{mountpoint:/var/lib/kubelet/pods/97872ff8-a1f5-4062-8202-41aa9d8a93bc/volumes/kubernetes.io~projected/kube-api-access-k2tfx major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b61b643-c3cf-453c-875e-25576445ad4f/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/9b61b643-c3cf-453c-875e-25576445ad4f/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b61b643-c3cf-453c-875e-25576445ad4f/volumes/kubernetes.io~projected/kube-api-access-rhvqf:{mountpoint:/var/lib/kubelet/pods/9b61b643-c3cf-453c-875e-25576445ad4f/volumes/kubernetes.io~projected/kube-api-access-rhvqf major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b61b643-c3cf-453c-875e-25576445ad4f/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/9b61b643-c3cf-453c-875e-25576445ad4f/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b61b643-c3cf-453c-875e-25576445ad4f/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/9b61b643-c3cf-453c-875e-25576445ad4f/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31/volumes/kubernetes.io~projected/kube-api-access-9b2ss:{mountpoint:/var/lib/kubelet/pods/a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31/volumes/kubernetes.io~projected/kube-api-access-9b2ss major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a66e14f7-3c72-48a5-88e8-e9f78a7c7ff5/volumes/kubernetes.io~projected/kube-api-access-h2mms:{mountpoint:/var/lib/kubelet/pods/a66e14f7-3c72-48a5-88e8-e9f78a7c7ff5/volumes/kubernetes.io~projected/kube-api-access-h2mms major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6/volumes/kubernetes.io~projected/kube-api-access-b67sr:{mountpoint:/var/lib/kubelet/pods/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6/volumes/kubernetes.io~projected/kube-api-access-b67sr major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4ef3fe7-a7fb-473c-a301-9073d502a86a/volumes/kubernetes.io~projected/kube-api-access-r9bws:{mountpoint:/var/lib/kubelet/pods/b4ef3fe7-a7fb-473c-a301-9073d502a86a/volumes/kubernetes.io~projected/kube-api-access-r9bws major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0b2deee-2b0c-4c2c-9ac5-debd33f12ba3/volumes/kubernetes.io~projected/kube-api-access-nxsnl:{mountpoint:/var/lib/kubelet/pods/c0b2deee-2b0c-4c2c-9ac5-debd33f12ba3/volumes/kubernetes.io~projected/kube-api-access-nxsnl major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c790e080-b546-409c-8ed2-1fc23982984b/volumes/kubernetes.io~projected/kube-api-access-6qzp2:{mountpoint:/var/lib/kubelet/pods/c790e080-b546-409c-8ed2-1fc23982984b/volumes/kubernetes.io~projected/kube-api-access-6qzp2 major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef2c01c8-12ef-4294-963e-4e840e2eecff/volumes/kubernetes.io~projected/kube-api-access-9hfkg:{mountpoint:/var/lib/kubelet/pods/ef2c01c8-12ef-4294-963e-4e840e2eecff/volumes/kubernetes.io~projected/kube-api-access-9hfkg major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f6337834-db43-4b8e-a955-9691f72e0f19/volumes/kubernetes.io~projected/kube-api-access-f7cv8:{mountpoint:/var/lib/kubelet/pods/f6337834-db43-4b8e-a955-9691f72e0f19/volumes/kubernetes.io~projected/kube-api-access-f7cv8 major:0 minor:60 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ea424199e685c6241f1f20898d96901aabc1632ed3b774fb241fe29e3f870bd1/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/ad3a0dfc65a3e368b367036bb75d1670a0c6430ba988fc6228f44910ebb4f0f2/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/539ffeba55d5619299fd32fc87a72a210536cef1d65006a6a0fb35f028ed8ce1/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/f226eed6d6f9c11bb190f1eede34cb147141e92f1afe1cac9dfa7d9eea3a6b31/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/ee4dddd80fd28d1a0ca342ceeecdd4f67efa5507bee55fd5fa4aa49b0105e4e4/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/6ddb89edf769398c2c85b430f8b39b852e7fe483d37e6ed3796b3cee683b38db/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/5fa75062fe396e25d85d16b3c7a2c5f7f4fc784889edf2fae90f223f389b9d66/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/363b5d31374d8ed38281e7e31eb74ea862a933e1155bb3809545893ead5e05b5/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/68f71e6c97d13586484e5ab1a86f1d8422a37802338efe5a6e27d84ceb24c756/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/3c07f1d5db05f385fd466f74149b54855048f1e3c404c4fa0f1c852b5ef4bc4e/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/e1d69450a050bbdbe45c681f2d13c33c2601bf0f1525efb456a5cec4f49bca0d/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/6bd620d81760b8c892f83de6fd6ecd58badadeabb6a70503e480244b4cbd5acd/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/9299483f7be5f1158729e3ba167edc80595c8f28d9ff8502f495ed2944559259/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/aebafd4d8ca0a191cad5186dec8f7fd74ee59bd013f95a3ae35f88b2bd118a03/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/1418a8f65be0bb6fdbd044e4f7966173541aeedfeb1c7e783c9c08fcd351b5f4/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/e7c5e4e5de8d8f65b15a8680332c154719b3ee5f2c9cdcf51ccac26d93f8af28/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/e5017716094d1247f6c42d430caac9f7edc068ba93b8c6cb5d31456076c372d4/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/2e205963bc1c42d13d9850970bdfa9f09c1eaaf78b5848aa3250118998b53fe3/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/7a6eec1dd941ae68492981d6a77d3f947c3b2aa263e9123e067dba125b75c3ea/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/84cc4b4cfe8f8b0bc07b9c5be7e2d33cf923894c2261e42fbde3a125c39d2ff6/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/d3a3772bce29c671c98df15527d15193eaa72cc2a9d213a982b5271f96d2aa6e/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/76b77d5987f94c4a372fa83f5cf4fbaf02732dc855e2067b6a14788772d429a7/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/7c9742ef39ad4a284a35226cb7a3eb1e62356332ccf712ecbf0f3e218403b5c5/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/209ddae04b6a744a59c6ee7dc2b110e382b8f8d5d4ac31d27febd98a6cd2e048/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/5ddc99f23d7d945496d1b6f67af6fb98fa5d73e53fed50c2be29783be5693fcc/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/a22f9df1c092383d788b494b07efa3cfc0839c019b01faf90d5a155e82febc25/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/5b30a678ecd5a2355a1aa9498f784353f4b0155038c07afb246c33e24f8ae51e/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/610a25b7523eca2b380cde317f166850fde42be8636954af155b644173dd28d4/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/cede0a673fc4cc14e085221678a2f5441fc886c6245686bbfc095c7e8e81b017/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/76a220f57c9e60d7ef8eb10b431e9a5f5f0d2639085a40609652622292bf720b/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/3a51a0e015eaaeb139c41b07eb672930c34e1c47f9271b0836caa32a6a9cc400/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/e81962e11ca305c6d84677432636ebb36ee1613c5c24d2cef7bbd9d2cc4be9bf/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/072a505b60c3ec50af0be9c59b46a9ab05677ff6474d8dc0984233c8920b33b3/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/3a93f2c09c7c899723bcc78746042bc98a2297c7839782e9063000e4b7898441/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/c6c9976b5eff182abca6106cf4352e222be7df697cf879dc368860b3d1126f6d/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/fce96f28f48c80c7173efd596a409c863ccb8e66c5e719570d1a5363c58ff9ec/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/075c9eb06206c439df36c410cd6ce4d4ed03a46e619fd236b1f427fcc2c3231c/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/981d0a972018d704d8022608a38a7fef63b8b7eb140989413f029629646e7f8b/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/61679f6c4420a7b3f143968798e8a8aa7c6dc45ee94b7daf59c81ccceafa7ac9/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/401897d2841d4bfc4c3b4cafa123b24028eb011c5cd4b2349efeddccdaef083d/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/173d46588d1370d2bd8479505451c9dbe1b118e64cc42974c9f8ed223905b4d9/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/349b432630d35b38f9d4972c53024647f25974a1464e3088f1e398cbc88eeab0/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/db28f408cb9ed022380a2ebd5ef5e7f2c916a0bb2cd1d75224ec2a281537a4db/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/410717349c82fc1c1ce6bc25c06fdfd330b75dfff9dd3bebdcfbe3bb74ff3cc2/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/385f1d2f6b1a5a7d99d9e59017a251a21194179fdd3861da834a5bd70431e61b/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/f15cae521eab71cec265b84d29a114ab590c12332b9da7c21834f39d6dd3c55b/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/a328cfc5794fd87b495b779f8be1046843e5a895f4499186d0cf4ce66f8c8e8f/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/3bb2ba87c222a1acf1dd80a8249c88c32134a2515a3daf5362279d7c8a4e2bd5/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/358875e977d1d05c3f9572df9c61b7cdcaa6faa53f6f2078b485ba536fb89c2f/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/88059ad5d80432483d9d39bd28d4a8b20a505e0b107085fd54e38608e271586d/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/2962c48c4bdc8ce0edcab883a42a49fe5813dfbde8c0bc1b607617a97c71178b/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/32fe8b78a3756ea468f0592ab32e80ab5052e0c3e88b16fe9b32f2daa28bd69e/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/0cce7bc2559d130f45b188f8265be5bbcc6de67aa9b3ad5e2b2204be2b026149/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/e03fcb8390440d79e940ef032cca4e8926b00ec10d315856b1f4ca540eab0ccf/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-95:{mountpoint:/var/lib/containers/storage/overlay/c00978e7ec4bbabb775c8ccde23ac639e273713eedab53d2dd257ed78ae3ab52/merged major:0 minor:95 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/bb978f6d28e30f5f6525d19d088668ab7fd521060ffdb3bbdd69b3734804ecaa/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 21:00:12.964658 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.964012 4926 manager.go:217] Machine: {Timestamp:2026-10-30 21:00:12.963552843 +0000 UTC m=+0.084626579 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200160 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22c8a44d470962e298a845cc7aa09c SystemUUID:ec22c8a4-4d47-0962-e298-a845cc7aa09c BootID:28d25c46-75df-4996-9cc7-b0820c83cb76 Filesystems:[{Device:overlay_0-95 DeviceMajor:0 DeviceMinor:95 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b4ba52c-ad04-4a25-9158-89941367abd3/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:221 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/662ead00-b524-4a6e-815b-c3399b3c9e6c/volumes/kubernetes.io~projected/kube-api-access-6tf9r DeviceMajor:0 DeviceMinor:226 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/705e2b0e-10e1-4710-8dfb-bbd862c8c084/volumes/kubernetes.io~projected/kube-api-access-x6gqs DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae4fa48a1d708a46447ff5ac1999bb738e3759a7b20711ec7e5b1472cc0a5b61/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7dd4d31803010cf19be81325287be8a522d556865e760858b16bfe0f4439421/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf089dbea2ba22737b1bb71e42206d667d7b745dcaa421f6d6abca101caa3ceb/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e63373561db72954e6ed55d6d4a214d939a57c30e5f981211acb86b0b653ab7f/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ae91fed0dc7feaa07ff952a02edbe8cd026ec5af13f2e6e31a2a30d5a9015e7/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c9e921e24c64a15435106455241e0ff8bfd38a668102ecd4b56609cc4ed29f10/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a2fb9ac-a905-4e20-b1be-7928d5e7881a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1ba5a3b6a76258cbeaf8a5c8af034efbdeb40e73ef0fe0493356c3d6b056133f/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f6337834-db43-4b8e-a955-9691f72e0f19/volumes/kubernetes.io~projected/kube-api-access-f7cv8 DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0b2deee-2b0c-4c2c-9ac5-debd33f12ba3/volumes/kubernetes.io~projected/kube-api-access-nxsnl DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b4ef3fe7-a7fb-473c-a301-9073d502a86a/volumes/kubernetes.io~projected/kube-api-access-r9bws DeviceMajor:0 DeviceMinor:312 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4961a255689d4f8b3ca3f78ac06b9076bfbd9659ab98826173df297fdb40eda9/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9fd2d7bbece44d55d7b897ad7783d27cac24ed24ae77eeed57b1a3de942d3cb5/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e1e8b14579a11d4ed7ed03b3a11a0ce6ebdd17b3bc316392fe600bb918be48c9/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5/volumes/kubernetes.io~projected/kube-api-access-dz6tt DeviceMajor:0 DeviceMinor:293 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/88f212cc-4eb8-4a49-9c13-a1d395ab43c6/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/6b4ba52c-ad04-4a25-9158-89941367abd3/volumes/kubernetes.io~projected/kube-api-access-vrt6b DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b61b643-c3cf-453c-875e-25576445ad4f/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3358db14-a01d-479e-a742-bdcd1ca80527/volumes/kubernetes.io~projected/kube-api-access-cgg8n DeviceMajor:0 DeviceMinor:217 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97872ff8-a1f5-4062-8202-41aa9d8a93bc/volumes/kubernetes.io~projected/kube-api-access-k2tfx DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/93f2c084-a1d2-4bda-8a41-c01829cc81ba/volumes/kubernetes.io~projected/kube-api-access-6g67v DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1a2fb9ac-a905-4e20-b1be-7928d5e7881a/volumes/kubernetes.io~projected/kube-api-access-pwf67 DeviceMajor:0 DeviceMinor:225 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9b61b643-c3cf-453c-875e-25576445ad4f/volumes/kubernetes.io~projected/kube-api-access-rhvqf DeviceMajor:0 DeviceMinor:223 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/ef2c01c8-12ef-4294-963e-4e840e2eecff/volumes/kubernetes.io~projected/kube-api-access-9hfkg DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e20348d-9970-4de7-80be-84f29ef8ece6/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6/volumes/kubernetes.io~projected/kube-api-access-b67sr DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0b77d4c7-37eb-4bad-8367-31346289726e/volumes/kubernetes.io~projected/kube-api-access-l2749 DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4901add7-3312-4950-ae4c-4d6581380e9e/volumes/kubernetes.io~projected/kube-api-access-4w859 DeviceMajor:0 DeviceMinor:227 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6b4ba52c-ad04-4a25-9158-89941367abd3/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:211 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5ae83504a960547e4f577294464b10264d64c8b625de5f99452e0f3ccda56293/userdata/shm DeviceMajor:0 DeviceMinor:334 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d5720d79d5bc05f502219a5b16458606d49b67752f9379196b789c214d91311/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d122169f6e92689276586e461c8d1c60c93d73f4f62ef18a38b79104e603723b/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d9daf948fdbabb2d3d1e892ca5c110c1242fbad1590c0fd045f06fb61391b932/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03dab2bd4f5ccd547d64bb7a5331444dfb48a537d0ebfa55136fc411976c629d/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b61b643-c3cf-453c-875e-25576445ad4f/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/635e766e79e31aed78f48ee33e9c38219337c8af31bced336183205b22c540ad/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/1e20348d-9970-4de7-80be-84f29ef8ece6/volumes/kubernetes.io~projected/kube-api-access-d58ss DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/93d288e4555cd3e2cc0345c8afcd8156f019e9f1443044bb18e3060667bd1079/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4901add7-3312-4950-ae4c-4d6581380e9e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6b4ba52c-ad04-4a25-9158-89941367abd3/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:216 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9b61b643-c3cf-453c-875e-25576445ad4f/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:224 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e20348d-9970-4de7-80be-84f29ef8ece6/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31/volumes/kubernetes.io~projected/kube-api-access-9b2ss DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4307f887bef1e1ad6367874facafc8cffd4c890d6415ff5c3dd10da77ae5b8f9/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0f0b2878-417e-4986-8067-b50a16d16967/volumes/kubernetes.io~projected/kube-api-access-z2k65 DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/6d9c7935-1d3d-4323-ac72-008a394e7f31/volumes/kubernetes.io~projected/kube-api-access-nmcw2 DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c790e080-b546-409c-8ed2-1fc23982984b/volumes/kubernetes.io~projected/kube-api-access-6qzp2 DeviceMajor:0 DeviceMinor:313 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/705e2b0e-10e1-4710-8dfb-bbd862c8c084/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fdbd72ddb43ee4578e925ba88e43c3fb79e43f8fa5709ce70d51039ec2502586/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/03e64eb3-e7df-464d-9382-47cd0e570456/volumes/kubernetes.io~projected/kube-api-access-9xcm4 DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/88f212cc-4eb8-4a49-9c13-a1d395ab43c6/volumes/kubernetes.io~projected/kube-api-access-67blk DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c67809ee48958159646ad059a290b0559a8e206498105424fc67d98b5537cd83/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a66e14f7-3c72-48a5-88e8-e9f78a7c7ff5/volumes/kubernetes.io~projected/kube-api-access-h2mms DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4ce43826-7550-4c62-8acc-48714541e426/volumes/kubernetes.io~projected/kube-api-access-gvl2l DeviceMajor:0 DeviceMinor:220 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/88f212cc-4eb8-4a49-9c13-a1d395ab43c6/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fcc5c3c5eb56c860c84b8d31c502139f42a747a5ad1b6e0105faf9366a948c44/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:209 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/81d54f22853849278d8b2c649c0d4a515ab479ecfeef0013099e3f3f5dc690c5/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1362c712-7332-4a6e-a215-633d2f02acca/volumes/kubernetes.io~projected/kube-api-access-dhhrx DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8bc258c0-5d93-4ddb-b49c-9a06167f6185/volumes/kubernetes.io~projected/kube-api-access-x799c DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4ce43826-7550-4c62-8acc-48714541e426/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:215 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d/volumes/kubernetes.io~projected/kube-api-access-cz752 DeviceMajor:0 DeviceMinor:222 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e19a827d04e419c8553f530ce0591dd5284d48acb9b1ae403d0ced5d7909c7ba/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1ba5a3b6a76258c MacAddress:ea:82:f7:eb:c0:02 Speed:10000 Mtu:8901} {Name:4307f887bef1e1a MacAddress:ce:d6:1a:2e:d7:78 Speed:10000 Mtu:8901} {Name:5ae83504a960547 MacAddress:fa:63:c3:95:06:49 Speed:10000 Mtu:8901} {Name:635e766e79e31ae MacAddress:46:f8:bd:39:78:ed Speed:10000 Mtu:8901} {Name:81d54f228538492 MacAddress:ea:64:23:52:fe:52 Speed:10000 Mtu:8901} {Name:93d288e4555cd3e MacAddress:b6:33:57:17:b3:26 Speed:10000 Mtu:8901} {Name:ae4fa48a1d708a4 MacAddress:5a:b7:5e:ea:f2:4d Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ee:a5:d9:ff Speed:0 Mtu:9001} {Name:br-int MacAddress:fe:59:9a:64:ac:4b Speed:0 Mtu:8901} {Name:d122169f6e92689 MacAddress:36:f1:53:70:fb:ed Speed:10000 Mtu:8901} {Name:d9daf948fdbabb2 MacAddress:de:aa:ae:20:3a:85 Speed:10000 Mtu:8901} {Name:e1e8b14579a11d4 MacAddress:fa:7a:4d:64:52:12 Speed:10000 Mtu:8901} {Name:e63373561db7295 MacAddress:12:db:5c:e3:6b:78 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ee:a5:d9:ff Speed:0 Mtu:9001} {Name:f7dd4d31803010c MacAddress:de:32:43:30:75:b4 Speed:10000 Mtu:8901} {Name:fdbd72ddb43ee45 MacAddress:12:69:a8:d1:b2:c6 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:5e:05:2c:9d:42:9b Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:56:d4:95:7b:80:84 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:00:12.964658 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.964381 4926 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:00:12.964658 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.964428 4926 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:00:12.964891 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.964722 4926 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:00:12.964891 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.964743 4926 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-3.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:00:12.964891 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.964877 4926 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:00:12.964891 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.964885 4926 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:00:12.965002 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.964898 4926 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:00:12.965002 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.964911 4926 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:00:12.965002 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.964929 4926 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:00:12.965112 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.965100 4926 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:00:12.965230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.965221 4926 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:00:12.965267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.965258 4926 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:00:12.965295 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.965275 4926 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:00:12.965295 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.965286 4926 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:00:12.965295 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.965294 4926 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:00:12.965393 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.965302 4926 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:00:12.966237 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.966216 4926 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:00:12.966237 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.966234 4926 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:00:12.966761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.966736 4926 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:00:12.969233 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.969219 4926 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:00:12.969233 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.969232 4926 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:00:12.969324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.969317 4926 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:00:12.969498 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.969487 4926 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:00:12.969542 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.969515 4926 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:00:12.969604 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.969596 4926 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:00:12.969648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.969610 4926 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:00:12.969648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.969616 4926 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:00:12.969648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.969621 4926 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:00:12.969648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.969627 4926 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:00:12.969648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.969633 4926 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:00:12.969648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.969639 4926 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:00:12.969648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.969644 4926 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:00:12.969648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.969650 4926 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:00:12.969920 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.969656 4926 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:00:12.969920 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.969664 4926 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:00:12.969920 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.969673 4926 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:00:12.969920 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.969695 4926 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:00:12.969920 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.969701 4926 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:00:12.973011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.972998 4926 server.go:1263] "Started kubelet" Oct 30 21:00:12.973098 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.973075 4926 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:00:12.973554 ip-10-0-142-3 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:00:12.976850 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.975286 4926 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:00:12.977149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.977133 4926 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:00:12.977608 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.977569 4926 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:00:12.977815 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.977798 4926 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:00:12.981591 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.980954 4926 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:00:12.981591 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.981068 4926 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:00:12.981591 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.981079 4926 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:00:12.981591 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.981097 4926 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:00:12.981591 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.981119 4926 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 20:54:34 +0000 UTC" deadline="2028-07-17 00:21:57.051933458 +0000 UTC" Oct 30 21:00:12.981591 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.981136 4926 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15003h21m44.070800565s" Oct 30 21:00:12.981591 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.981151 4926 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:00:12.981591 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:12.981185 4926 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-3.ec2.internal\" not found" Oct 30 21:00:12.981992 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.981932 4926 factory.go:55] Registering systemd factory Oct 30 21:00:12.981992 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.981950 4926 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:00:12.982340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.982325 4926 factory.go:153] Registering CRI-O factory Oct 30 21:00:12.982340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.982341 4926 factory.go:223] Registration of the crio container factory successfully Oct 30 21:00:12.982479 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.982390 4926 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:00:12.982479 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.982410 4926 factory.go:103] Registering Raw factory Oct 30 21:00:12.982479 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.982424 4926 manager.go:1199] Started watching for new ooms in manager Oct 30 21:00:12.983528 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.982762 4926 manager.go:319] Starting recovery of all containers Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984410 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bc258c0-5d93-4ddb-b49c-9a06167f6185" volumeName="kubernetes.io/configmap/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-daemon-config" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984460 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b4ba52c-ad04-4a25-9158-89941367abd3" volumeName="kubernetes.io/secret/6b4ba52c-ad04-4a25-9158-89941367abd3-installation-pull-secrets" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984474 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6" volumeName="kubernetes.io/empty-dir/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-tmp" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984485 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="276d4076-cd5a-45d6-b4f8-2de14f9c9a0d" volumeName="kubernetes.io/projected/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-kube-api-access-cz752" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984498 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4901add7-3312-4950-ae4c-4d6581380e9e" volumeName="kubernetes.io/projected/4901add7-3312-4950-ae4c-4d6581380e9e-kube-api-access-4w859" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984509 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="705e2b0e-10e1-4710-8dfb-bbd862c8c084" volumeName="kubernetes.io/secret/705e2b0e-10e1-4710-8dfb-bbd862c8c084-serving-cert" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984520 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1362c712-7332-4a6e-a215-633d2f02acca" volumeName="kubernetes.io/configmap/1362c712-7332-4a6e-a215-633d2f02acca-serviceca" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984532 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88f212cc-4eb8-4a49-9c13-a1d395ab43c6" volumeName="kubernetes.io/configmap/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984544 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88f212cc-4eb8-4a49-9c13-a1d395ab43c6" volumeName="kubernetes.io/secret/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984554 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31" volumeName="kubernetes.io/empty-dir/a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31-tmp-dir" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984566 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31" volumeName="kubernetes.io/projected/a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31-kube-api-access-9b2ss" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984577 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0f0b2878-417e-4986-8067-b50a16d16967" volumeName="kubernetes.io/projected/0f0b2878-417e-4986-8067-b50a16d16967-kube-api-access-z2k65" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984588 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4901add7-3312-4950-ae4c-4d6581380e9e" volumeName="kubernetes.io/secret/4901add7-3312-4950-ae4c-4d6581380e9e-cert" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984600 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88f212cc-4eb8-4a49-9c13-a1d395ab43c6" volumeName="kubernetes.io/configmap/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-env-overrides" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984621 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b61b643-c3cf-453c-875e-25576445ad4f" volumeName="kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-kube-api-access-rhvqf" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984635 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ce43826-7550-4c62-8acc-48714541e426" volumeName="kubernetes.io/configmap/4ce43826-7550-4c62-8acc-48714541e426-config" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984648 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ce43826-7550-4c62-8acc-48714541e426" volumeName="kubernetes.io/secret/4ce43826-7550-4c62-8acc-48714541e426-serving-cert" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984661 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b4ba52c-ad04-4a25-9158-89941367abd3" volumeName="kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-bound-sa-token" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984671 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88f212cc-4eb8-4a49-9c13-a1d395ab43c6" volumeName="kubernetes.io/projected/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-kube-api-access-67blk" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984681 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ffe8e19-6888-4eca-ae9d-9d0fc05ba752" volumeName="kubernetes.io/configmap/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-nginx-conf" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984691 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c790e080-b546-409c-8ed2-1fc23982984b" volumeName="kubernetes.io/projected/c790e080-b546-409c-8ed2-1fc23982984b-kube-api-access-6qzp2" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984700 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88f212cc-4eb8-4a49-9c13-a1d395ab43c6" volumeName="kubernetes.io/configmap/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-ovnkube-config" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984711 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93f2c084-a1d2-4bda-8a41-c01829cc81ba" volumeName="kubernetes.io/projected/93f2c084-a1d2-4bda-8a41-c01829cc81ba-kube-api-access-6g67v" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984722 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e20348d-9970-4de7-80be-84f29ef8ece6" volumeName="kubernetes.io/empty-dir/1e20348d-9970-4de7-80be-84f29ef8ece6-tmp" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984732 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03e64eb3-e7df-464d-9382-47cd0e570456" volumeName="kubernetes.io/configmap/03e64eb3-e7df-464d-9382-47cd0e570456-cni-binary-copy" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984742 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e20348d-9970-4de7-80be-84f29ef8ece6" volumeName="kubernetes.io/empty-dir/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-tuned" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984753 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6" volumeName="kubernetes.io/projected/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-kube-api-access-b67sr" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984770 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b61b643-c3cf-453c-875e-25576445ad4f" volumeName="kubernetes.io/empty-dir/9b61b643-c3cf-453c-875e-25576445ad4f-ca-trust-extracted" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984787 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b61b643-c3cf-453c-875e-25576445ad4f" volumeName="kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-bound-sa-token" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984799 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef2c01c8-12ef-4294-963e-4e840e2eecff" volumeName="kubernetes.io/configmap/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-config" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984812 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03e64eb3-e7df-464d-9382-47cd0e570456" volumeName="kubernetes.io/configmap/03e64eb3-e7df-464d-9382-47cd0e570456-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984825 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b4ba52c-ad04-4a25-9158-89941367abd3" volumeName="kubernetes.io/configmap/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-certificates" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984836 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bc258c0-5d93-4ddb-b49c-9a06167f6185" volumeName="kubernetes.io/projected/8bc258c0-5d93-4ddb-b49c-9a06167f6185-kube-api-access-x799c" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984849 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4ef3fe7-a7fb-473c-a301-9073d502a86a" volumeName="kubernetes.io/empty-dir/b4ef3fe7-a7fb-473c-a301-9073d502a86a-data-volume" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984862 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2fb9ac-a905-4e20-b1be-7928d5e7881a" volumeName="kubernetes.io/configmap/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-trusted-ca" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984874 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2fb9ac-a905-4e20-b1be-7928d5e7881a" volumeName="kubernetes.io/projected/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-kube-api-access-pwf67" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984887 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b61b643-c3cf-453c-875e-25576445ad4f" volumeName="kubernetes.io/secret/9b61b643-c3cf-453c-875e-25576445ad4f-image-registry-private-configuration" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984900 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f6337834-db43-4b8e-a955-9691f72e0f19" volumeName="kubernetes.io/projected/f6337834-db43-4b8e-a955-9691f72e0f19-kube-api-access-f7cv8" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984912 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e" volumeName="kubernetes.io/configmap/5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e-konnectivity-ca" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984924 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="662ead00-b524-4a6e-815b-c3399b3c9e6c" volumeName="kubernetes.io/configmap/662ead00-b524-4a6e-815b-c3399b3c9e6c-config-volume" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984937 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6" volumeName="kubernetes.io/secret/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-serving-cert" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984950 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1362c712-7332-4a6e-a215-633d2f02acca" volumeName="kubernetes.io/projected/1362c712-7332-4a6e-a215-633d2f02acca-kube-api-access-dhhrx" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984962 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="276d4076-cd5a-45d6-b4f8-2de14f9c9a0d" volumeName="kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-default-certificate" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984975 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e" volumeName="kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.984986 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="662ead00-b524-4a6e-815b-c3399b3c9e6c" volumeName="kubernetes.io/projected/662ead00-b524-4a6e-815b-c3399b3c9e6c-kube-api-access-6tf9r" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985027 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6" volumeName="kubernetes.io/configmap/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-service-ca-bundle" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985057 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6" volumeName="kubernetes.io/empty-dir/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-snapshots" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985070 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4ef3fe7-a7fb-473c-a301-9073d502a86a" volumeName="kubernetes.io/projected/b4ef3fe7-a7fb-473c-a301-9073d502a86a-kube-api-access-r9bws" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985084 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b61b643-c3cf-453c-875e-25576445ad4f" volumeName="kubernetes.io/configmap/9b61b643-c3cf-453c-875e-25576445ad4f-trusted-ca" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985097 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e20348d-9970-4de7-80be-84f29ef8ece6" volumeName="kubernetes.io/projected/1e20348d-9970-4de7-80be-84f29ef8ece6-kube-api-access-d58ss" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985111 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4ce43826-7550-4c62-8acc-48714541e426" volumeName="kubernetes.io/projected/4ce43826-7550-4c62-8acc-48714541e426-kube-api-access-gvl2l" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985124 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e" volumeName="kubernetes.io/secret/5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e-agent-certs" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985138 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b4ba52c-ad04-4a25-9158-89941367abd3" volumeName="kubernetes.io/empty-dir/6b4ba52c-ad04-4a25-9158-89941367abd3-ca-trust-extracted" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985150 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b4ba52c-ad04-4a25-9158-89941367abd3" volumeName="kubernetes.io/secret/6b4ba52c-ad04-4a25-9158-89941367abd3-image-registry-private-configuration" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985163 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97872ff8-a1f5-4062-8202-41aa9d8a93bc" volumeName="kubernetes.io/projected/97872ff8-a1f5-4062-8202-41aa9d8a93bc-kube-api-access-k2tfx" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985174 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b61b643-c3cf-453c-875e-25576445ad4f" volumeName="kubernetes.io/secret/9b61b643-c3cf-453c-875e-25576445ad4f-installation-pull-secrets" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985194 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3358db14-a01d-479e-a742-bdcd1ca80527" volumeName="kubernetes.io/projected/3358db14-a01d-479e-a742-bdcd1ca80527-kube-api-access-cgg8n" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985207 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e46662c-bfe6-45f8-8a23-89ea5fc92ae5" volumeName="kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985220 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b61b643-c3cf-453c-875e-25576445ad4f" volumeName="kubernetes.io/configmap/9b61b643-c3cf-453c-875e-25576445ad4f-registry-certificates" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985232 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6" volumeName="kubernetes.io/configmap/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-trusted-ca-bundle" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985243 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef2c01c8-12ef-4294-963e-4e840e2eecff" volumeName="kubernetes.io/projected/ef2c01c8-12ef-4294-963e-4e840e2eecff-kube-api-access-9hfkg" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985256 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8bc258c0-5d93-4ddb-b49c-9a06167f6185" volumeName="kubernetes.io/configmap/8bc258c0-5d93-4ddb-b49c-9a06167f6185-cni-binary-copy" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985268 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97872ff8-a1f5-4062-8202-41aa9d8a93bc" volumeName="kubernetes.io/configmap/97872ff8-a1f5-4062-8202-41aa9d8a93bc-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985280 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03e64eb3-e7df-464d-9382-47cd0e570456" volumeName="kubernetes.io/projected/03e64eb3-e7df-464d-9382-47cd0e570456-kube-api-access-9xcm4" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985296 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b77d4c7-37eb-4bad-8367-31346289726e" volumeName="kubernetes.io/projected/0b77d4c7-37eb-4bad-8367-31346289726e-kube-api-access-l2749" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985320 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="662ead00-b524-4a6e-815b-c3399b3c9e6c" volumeName="kubernetes.io/empty-dir/662ead00-b524-4a6e-815b-c3399b3c9e6c-tmp-dir" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985331 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d9c7935-1d3d-4323-ac72-008a394e7f31" volumeName="kubernetes.io/projected/6d9c7935-1d3d-4323-ac72-008a394e7f31-kube-api-access-nmcw2" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985346 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="705e2b0e-10e1-4710-8dfb-bbd862c8c084" volumeName="kubernetes.io/projected/705e2b0e-10e1-4710-8dfb-bbd862c8c084-kube-api-access-x6gqs" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985360 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0b2deee-2b0c-4c2c-9ac5-debd33f12ba3" volumeName="kubernetes.io/projected/c0b2deee-2b0c-4c2c-9ac5-debd33f12ba3-kube-api-access-nxsnl" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985372 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2fb9ac-a905-4e20-b1be-7928d5e7881a" volumeName="kubernetes.io/configmap/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-config" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985386 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a66e14f7-3c72-48a5-88e8-e9f78a7c7ff5" volumeName="kubernetes.io/projected/a66e14f7-3c72-48a5-88e8-e9f78a7c7ff5-kube-api-access-h2mms" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985399 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03e64eb3-e7df-464d-9382-47cd0e570456" volumeName="kubernetes.io/configmap/03e64eb3-e7df-464d-9382-47cd0e570456-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985412 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b77d4c7-37eb-4bad-8367-31346289726e" volumeName="kubernetes.io/configmap/0b77d4c7-37eb-4bad-8367-31346289726e-telemetry-config" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985425 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a2fb9ac-a905-4e20-b1be-7928d5e7881a" volumeName="kubernetes.io/secret/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-serving-cert" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985438 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="276d4076-cd5a-45d6-b4f8-2de14f9c9a0d" volumeName="kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-stats-auth" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985449 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b4ba52c-ad04-4a25-9158-89941367abd3" volumeName="kubernetes.io/configmap/6b4ba52c-ad04-4a25-9158-89941367abd3-trusted-ca" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985462 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6b4ba52c-ad04-4a25-9158-89941367abd3" volumeName="kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-kube-api-access-vrt6b" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985473 4926 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="705e2b0e-10e1-4710-8dfb-bbd862c8c084" volumeName="kubernetes.io/configmap/705e2b0e-10e1-4710-8dfb-bbd862c8c084-config" seLinuxMountContext="" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985485 4926 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:00:12.985496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.985493 4926 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:00:12.997400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:12.997370 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:00:13.000462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.000441 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:00:13.006485 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.006434 4926 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:00:13.007656 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.007635 4926 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:00:13.007656 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.007655 4926 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:00:13.007780 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.007675 4926 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:00:13.007780 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:13.007715 4926 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:00:13.020935 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.020907 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:00:13.021060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.020946 4926 generic.go:356] "Generic (PLEG): container finished" podID="1a2fb9ac-a905-4e20-b1be-7928d5e7881a" containerID="cb966d4ceaaab215d2d47172a0b5244daddd96c1007b782eaaa1ed2d0d47b7ee" exitCode=255 Oct 30 21:00:13.022609 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.022588 4926 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:00:13.022719 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.022703 4926 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:00:13.023538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.023509 4926 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:00:13.034741 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.034718 4926 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:00:13.034849 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.034800 4926 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:00:13.053598 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.053564 4926 generic.go:356] "Generic (PLEG): container finished" podID="03e64eb3-e7df-464d-9382-47cd0e570456" containerID="909bdfaa047b2ffe9cf2079a75ea39471b3de7ea9bf8fd5c7653f8c06ec5cec1" exitCode=0 Oct 30 21:00:13.053598 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.053592 4926 generic.go:356] "Generic (PLEG): container finished" podID="03e64eb3-e7df-464d-9382-47cd0e570456" containerID="840016763d62b059579856ba431dbd0df93a21a1c5c14715b6d7b015ae36383d" exitCode=0 Oct 30 21:00:13.053598 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.053605 4926 generic.go:356] "Generic (PLEG): container finished" podID="03e64eb3-e7df-464d-9382-47cd0e570456" containerID="2e9016d861ab1a8ba80f2f74331308204bb48fe7ff770da101c1570bee7eb698" exitCode=0 Oct 30 21:00:13.053820 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.053612 4926 generic.go:356] "Generic (PLEG): container finished" podID="03e64eb3-e7df-464d-9382-47cd0e570456" containerID="4c2220296aeccf5ffcd6081399bb63ad9567023d8ed635ad356eaffb71383ff1" exitCode=0 Oct 30 21:00:13.053820 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.053617 4926 generic.go:356] "Generic (PLEG): container finished" podID="03e64eb3-e7df-464d-9382-47cd0e570456" containerID="8c66728984e57caf7e21a9a9cf706207b308ea9a73b6db26767afd57f87d5ce7" exitCode=0 Oct 30 21:00:13.053820 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.053622 4926 generic.go:356] "Generic (PLEG): container finished" podID="03e64eb3-e7df-464d-9382-47cd0e570456" containerID="7d4d7e8045f383fb54cd7a6b7dce08070023d6b4df2a9698c17f158f5ad9407e" exitCode=0 Oct 30 21:00:13.055514 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.055493 4926 generic.go:356] "Generic (PLEG): container finished" podID="86fd282564aec729bf61e75aa114744a" containerID="3ed8e2409579bd986770f5c33540d4899bae49deadbd613baab5f08db05729cd" exitCode=0 Oct 30 21:00:13.073995 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.073937 4926 manager.go:324] Recovery completed Oct 30 21:00:13.091974 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.091953 4926 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:00:13.091974 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.091967 4926 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:00:13.092109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.091984 4926 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:00:13.092139 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.092111 4926 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:00:13.092139 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.092121 4926 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:00:13.092139 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.092138 4926 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:00:13.092236 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.092143 4926 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:00:13.092236 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.092148 4926 policy_none.go:50] "Start" Oct 30 21:00:13.092236 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.092156 4926 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:00:13.092236 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.092165 4926 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:00:13.092336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.092304 4926 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:00:13.092336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.092311 4926 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:00:13.092336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.092317 4926 policy_none.go:44] "Start" Oct 30 21:00:13.096479 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.096457 4926 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:00:13.098984 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.098973 4926 manager.go:341] "Starting Device Plugin manager" Oct 30 21:00:13.099673 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:13.098998 4926 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:00:13.099731 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.099678 4926 server.go:87] "Starting device plugin registration server" Oct 30 21:00:13.099992 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.099979 4926 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:00:13.100025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.099998 4926 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:00:13.100101 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.100087 4926 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:00:13.100188 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.100159 4926 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:00:13.100188 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.100190 4926 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:00:13.100360 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.100196 4926 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:00:13.100360 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.100242 4926 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:00:13.100180422Z","UUID":"4e3bea55-e4b0-4c77-85b9-0841917ad347","Handler":null,"Name":"","Endpoint":""} Oct 30 21:00:13.101794 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.101772 4926 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:00:13.101888 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.101813 4926 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:00:13.101951 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:13.101895 4926 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:00:13.107798 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.107753 4926 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-3.ec2.internal"] Oct 30 21:00:13.108435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.108405 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" event={"ID":"86fd282564aec729bf61e75aa114744a","Type":"ContainerStarted","Data":"1beab69337c9ae8002f086417000c8864d06ef0bcd0c69d8141f51b0cdef4f5f"} Oct 30 21:00:13.108511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.108444 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" event={"ID":"86fd282564aec729bf61e75aa114744a","Type":"ContainerDied","Data":"3ed8e2409579bd986770f5c33540d4899bae49deadbd613baab5f08db05729cd"} Oct 30 21:00:13.108511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.108466 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" event={"ID":"86fd282564aec729bf61e75aa114744a","Type":"ContainerStarted","Data":"8ae91fed0dc7feaa07ff952a02edbe8cd026ec5af13f2e6e31a2a30d5a9015e7"} Oct 30 21:00:13.108511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.108491 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-3.ec2.internal" event={"ID":"10ed65d1de0a53fd4cc159ce5efb836d","Type":"ContainerStarted","Data":"e6515524026e82ce86fcb69cc026ff764b2ab091e9e8a238f2aaf15ab91f86e2"} Oct 30 21:00:13.108511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.108508 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-3.ec2.internal" event={"ID":"10ed65d1de0a53fd4cc159ce5efb836d","Type":"ContainerStarted","Data":"9d5720d79d5bc05f502219a5b16458606d49b67752f9379196b789c214d91311"} Oct 30 21:00:13.108680 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.108444 4926 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" Oct 30 21:00:13.108680 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.108626 4926 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-3.ec2.internal" Oct 30 21:00:13.132487 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.132470 4926 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:00:13.132569 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:13.132507 4926 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-3.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-3.ec2.internal" Oct 30 21:00:13.133185 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.133172 4926 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:00:13.133222 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:13.133206 4926 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" Oct 30 21:00:13.201095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.201067 4926 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:00:13.202305 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.202279 4926 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasSufficientMemory" Oct 30 21:00:13.202413 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.202318 4926 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 21:00:13.202413 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.202329 4926 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-3.ec2.internal" event="NodeHasSufficientPID" Oct 30 21:00:13.202413 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.202370 4926 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-3.ec2.internal" Oct 30 21:00:13.223262 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.223237 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/86fd282564aec729bf61e75aa114744a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal\" (UID: \"86fd282564aec729bf61e75aa114744a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" Oct 30 21:00:13.223262 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.223262 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86fd282564aec729bf61e75aa114744a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal\" (UID: \"86fd282564aec729bf61e75aa114744a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" Oct 30 21:00:13.223412 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.223280 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10ed65d1de0a53fd4cc159ce5efb836d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-3.ec2.internal\" (UID: \"10ed65d1de0a53fd4cc159ce5efb836d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-3.ec2.internal" Oct 30 21:00:13.233759 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.233739 4926 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-3.ec2.internal" Oct 30 21:00:13.233854 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.233802 4926 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-3.ec2.internal" Oct 30 21:00:13.324448 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.324356 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/86fd282564aec729bf61e75aa114744a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal\" (UID: \"86fd282564aec729bf61e75aa114744a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" Oct 30 21:00:13.324448 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.324372 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/86fd282564aec729bf61e75aa114744a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal\" (UID: \"86fd282564aec729bf61e75aa114744a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" Oct 30 21:00:13.324448 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.324441 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86fd282564aec729bf61e75aa114744a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal\" (UID: \"86fd282564aec729bf61e75aa114744a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" Oct 30 21:00:13.324721 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.324468 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10ed65d1de0a53fd4cc159ce5efb836d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-3.ec2.internal\" (UID: \"10ed65d1de0a53fd4cc159ce5efb836d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-3.ec2.internal" Oct 30 21:00:13.324721 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.324516 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/10ed65d1de0a53fd4cc159ce5efb836d-config\") pod \"kube-apiserver-proxy-ip-10-0-142-3.ec2.internal\" (UID: \"10ed65d1de0a53fd4cc159ce5efb836d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-3.ec2.internal" Oct 30 21:00:13.324721 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.324561 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/86fd282564aec729bf61e75aa114744a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal\" (UID: \"86fd282564aec729bf61e75aa114744a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" Oct 30 21:00:13.965763 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:13.965733 4926 apiserver.go:52] "Watching apiserver" Oct 30 21:00:14.016569 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.016540 4926 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:00:14.017206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.017181 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-758fcd4795-j6bts","openshift-insights/insights-operator-7c4986d449-m7m4c","openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm","openshift-image-registry/image-registry-56886949f8-xwhft","openshift-image-registry/node-ca-cnj4q","openshift-ingress/router-default-6789dbc4b5-5gqgl","kube-system/kas-connection-checker-685895c7c9-zr75n","openshift-multus/network-metrics-daemon-7d6gs","openshift-kube-storage-version-migrator/migrator-66df5d496b-5v597","openshift-multus/multus-ggbwl","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87c42","openshift-cluster-node-tuning-operator/tuned-rkcv2","openshift-dns/dns-default-888xx","openshift-ingress-canary/ingress-canary-mmzzs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal","openshift-network-operator/iptables-alerter-ww5pw","kube-system/kas-connection-checker-685895c7c9-b9fhn","openshift-multus/multus-additional-cni-plugins-9gv9s","openshift-network-diagnostics/network-check-target-mfgjb","openshift-console-operator/console-operator-7cd56b8dbc-c5khs","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7","openshift-network-diagnostics/network-check-source-6f78d59f99-j5tcr","kube-system/kas-connection-checker-685895c7c9-zdvb2","openshift-insights/insights-runtime-extractor-q9bnp","openshift-dns/node-resolver-54qxj","kube-system/konnectivity-agent-mgsr5","kube-system/kube-apiserver-proxy-ip-10-0-142-3.ec2.internal","openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz","openshift-ovn-kubernetes/ovnkube-node-vtgj9","openshift-network-console/networking-console-plugin-86c478bb88-wbrhm","kube-system/global-pull-secret-syncer-m9dm5"] Oct 30 21:00:14.019734 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.019710 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:14.020717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.020701 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:14.021105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.021088 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" Oct 30 21:00:14.021484 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.021470 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:14.021874 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.021859 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" Oct 30 21:00:14.028676 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.028655 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 21:00:14.030905 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.030882 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:14.031366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.031352 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-888xx" Oct 30 21:00:14.032435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.032418 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:14.032515 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.032483 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 21:00:14.032515 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.032482 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 21:00:14.032843 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.032824 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 21:00:14.033239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.033216 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 21:00:14.033313 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.033255 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:00:14.033464 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.033447 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 21:00:14.033645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.033622 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 21:00:14.033645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.033626 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:00:14.033835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.033635 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:00:14.033835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.033659 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:00:14.033835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.033688 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 21:00:14.033835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.033734 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 21:00:14.033835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.033762 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 21:00:14.034066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.033893 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 21:00:14.034066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.034000 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-blkcc\"" Oct 30 21:00:14.034376 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.034363 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 21:00:14.034435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.034427 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-jn4qp\"" Oct 30 21:00:14.034502 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.034490 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-kmr6z\"" Oct 30 21:00:14.044778 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.044759 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:00:14.044884 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.044812 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-bkjxf\"" Oct 30 21:00:14.044884 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.044820 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:00:14.044884 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.044820 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:00:14.044884 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.044826 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:00:14.044884 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.044877 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:00:14.045142 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.044811 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:00:14.045142 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.044931 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-f7r6m\"" Oct 30 21:00:14.045142 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.044933 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:00:14.045142 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.044980 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:00:14.045142 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.045025 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:00:14.045142 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.045044 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 21:00:14.045142 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.045070 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-q5gbc\"" Oct 30 21:00:14.045350 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.045148 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:00:14.045350 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.045251 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:00:14.045961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.045945 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 21:00:14.046346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.046329 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:00:14.046925 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.046912 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-hvtl2\"" Oct 30 21:00:14.047290 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.047274 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:00:14.047375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.047279 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 21:00:14.047375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.047311 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b8vzq\"" Oct 30 21:00:14.047588 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.047573 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:00:14.047649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.047629 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5jdx7\"" Oct 30 21:00:14.047702 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.047665 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4x2zw\"" Oct 30 21:00:14.047702 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.047687 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 21:00:14.047702 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.047687 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fz2bf\"" Oct 30 21:00:14.048018 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048002 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-79n9v\"" Oct 30 21:00:14.048124 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048084 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-56hzc\"" Oct 30 21:00:14.048124 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048099 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:00:14.048211 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048141 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j8v8x\"" Oct 30 21:00:14.048211 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048162 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:00:14.048400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048351 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:00:14.048400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048390 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:00:14.048400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048355 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:00:14.048547 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048372 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 21:00:14.048547 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048365 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 21:00:14.048547 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048388 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:00:14.048547 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048390 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 21:00:14.048547 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048355 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:00:14.048547 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048445 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:00:14.048547 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048501 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:00:14.048772 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048732 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:00:14.048820 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048785 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jljmv\"" Oct 30 21:00:14.048820 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048807 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:00:14.048820 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048814 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:00:14.048962 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048833 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:00:14.048962 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048834 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-tzc2p\"" Oct 30 21:00:14.048962 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048866 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gszwc\"" Oct 30 21:00:14.048962 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048913 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 21:00:14.048962 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048921 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:00:14.048962 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048932 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 21:00:14.048962 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.048960 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-4k2gt\"" Oct 30 21:00:14.049236 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.049002 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 21:00:14.049672 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.049653 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:00:14.049750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.049694 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:00:14.049750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.049716 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:00:14.049866 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.049852 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:00:14.049916 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.049869 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:00:14.049977 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.049959 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:00:14.050622 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.050603 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:00:14.051128 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.051108 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:00:14.051233 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.051218 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 21:00:14.051323 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.051306 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 21:00:14.051381 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.051347 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:00:14.051427 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.051228 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:00:14.051505 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.051490 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4cglt\"" Oct 30 21:00:14.055917 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.055864 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 21:00:14.055917 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.055910 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 21:00:14.056273 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.056255 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:00:14.056373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.056265 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-tnqjk\"" Oct 30 21:00:14.056373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.056270 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:00:14.056373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.056340 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:00:14.056373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.056345 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:00:14.056373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.056318 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:00:14.056373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.056336 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:00:14.056610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.056379 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:00:14.056610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.056387 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:00:14.056610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.056317 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:00:14.056610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.056337 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:00:14.056610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.056313 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:00:14.056610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.056455 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-nnb2b\"" Oct 30 21:00:14.056610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.056468 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 21:00:14.056610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.056484 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xstbj\"" Oct 30 21:00:14.056610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.056546 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:00:14.056881 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.056685 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-p9dq9\"" Oct 30 21:00:14.056881 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.056718 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-fnzdx\"" Oct 30 21:00:14.056881 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.056782 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:00:14.056881 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.056818 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l57v4\"" Oct 30 21:00:14.056881 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.056871 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-bxqk5\"" Oct 30 21:00:14.057062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.056961 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mttl5\"" Oct 30 21:00:14.057500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.057487 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 21:00:14.063377 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.063361 4926 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-3.ec2.internal" Oct 30 21:00:14.063490 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.063477 4926 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" Oct 30 21:00:14.083463 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.083444 4926 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:00:14.086241 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.086228 4926 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:00:14.086289 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.086270 4926 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-3.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-3.ec2.internal" Oct 30 21:00:14.087100 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.087086 4926 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:00:14.087167 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.087126 4926 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-3.ec2.internal" Oct 30 21:00:14.128045 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128014 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz752\" (UniqueName: \"kubernetes.io/projected/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-kube-api-access-cz752\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:14.128130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128056 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-run-k8s-cni-cncf-io\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.128130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128073 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-run-netns\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.128200 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128129 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-run-openvswitch\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.128236 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128221 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-node-log\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.128266 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128250 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvl2l\" (UniqueName: \"kubernetes.io/projected/4ce43826-7550-4c62-8acc-48714541e426-kube-api-access-gvl2l\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-frsk7\" (UID: \"4ce43826-7550-4c62-8acc-48714541e426\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7" Oct 30 21:00:14.128311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128269 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/705e2b0e-10e1-4710-8dfb-bbd862c8c084-serving-cert\") pod \"service-ca-operator-544d749c8c-vprbm\" (UID: \"705e2b0e-10e1-4710-8dfb-bbd862c8c084\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm" Oct 30 21:00:14.128311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128286 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-var-lib-cni-bin\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.128402 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128310 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1362c712-7332-4a6e-a215-633d2f02acca-host\") pod \"node-ca-cnj4q\" (UID: \"1362c712-7332-4a6e-a215-633d2f02acca\") " pod="openshift-image-registry/node-ca-cnj4q" Oct 30 21:00:14.128402 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128352 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-bound-sa-token\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:14.128492 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128408 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-sysctl-d\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.128492 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128437 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-device-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 21:00:14.128492 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128462 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-service-ca-bundle\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:14.128637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128500 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-run-ovn\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.128637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128541 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b4ba52c-ad04-4a25-9158-89941367abd3-ca-trust-extracted\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:14.128637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128577 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b2ss\" (UniqueName: \"kubernetes.io/projected/a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31-kube-api-access-9b2ss\") pod \"node-resolver-54qxj\" (UID: \"a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31\") " pod="openshift-dns/node-resolver-54qxj" Oct 30 21:00:14.128637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128607 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5r7xx\" (UID: \"ef2c01c8-12ef-4294-963e-4e840e2eecff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" Oct 30 21:00:14.128637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128636 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-kubelet\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.128862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128646 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b4ba52c-ad04-4a25-9158-89941367abd3-ca-trust-extracted\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:14.128862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128655 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-kubernetes\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.128862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128658 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/705e2b0e-10e1-4710-8dfb-bbd862c8c084-serving-cert\") pod \"service-ca-operator-544d749c8c-vprbm\" (UID: \"705e2b0e-10e1-4710-8dfb-bbd862c8c084\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm" Oct 30 21:00:14.128862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128699 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 21:00:14.128862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128728 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0b77d4c7-37eb-4bad-8367-31346289726e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-v5rrz\" (UID: \"0b77d4c7-37eb-4bad-8367-31346289726e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:14.128862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128755 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:14.128862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128777 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-config\") pod \"console-operator-7cd56b8dbc-c5khs\" (UID: \"1a2fb9ac-a905-4e20-b1be-7928d5e7881a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:14.128862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128800 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e-agent-certs\") pod \"konnectivity-agent-mgsr5\" (UID: \"5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e\") " pod="kube-system/konnectivity-agent-mgsr5" Oct 30 21:00:14.128862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128824 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz6tt\" (UniqueName: \"kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt\") pod \"network-check-target-mfgjb\" (UID: \"4e46662c-bfe6-45f8-8a23-89ea5fc92ae5\") " pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 21:00:14.128862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128845 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-run-systemd\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.128862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128860 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgg8n\" (UniqueName: \"kubernetes.io/projected/3358db14-a01d-479e-a742-bdcd1ca80527-kube-api-access-cgg8n\") pod \"network-check-source-6f78d59f99-j5tcr\" (UID: \"3358db14-a01d-479e-a742-bdcd1ca80527\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j5tcr" Oct 30 21:00:14.129336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128879 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-original-pull-secret\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 21:00:14.129336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128895 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-systemd\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.129336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128920 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-sys\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.129336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128943 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-tuned\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.129336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128966 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03e64eb3-e7df-464d-9382-47cd0e570456-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 21:00:14.129336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128985 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-config\") pod \"console-operator-7cd56b8dbc-c5khs\" (UID: \"1a2fb9ac-a905-4e20-b1be-7928d5e7881a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:14.129336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.128990 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-run-netns\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.129336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129011 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0b77d4c7-37eb-4bad-8367-31346289726e-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-v5rrz\" (UID: \"0b77d4c7-37eb-4bad-8367-31346289726e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:14.129336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129024 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-var-lib-kubelet\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.129336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129075 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-lib-modules\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.129336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129112 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-service-ca-bundle\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:14.129336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129122 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-tuned\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.129336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129162 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2k65\" (UniqueName: \"kubernetes.io/projected/0f0b2878-417e-4986-8067-b50a16d16967-kube-api-access-z2k65\") pod \"kas-connection-checker-685895c7c9-zdvb2\" (UID: \"0f0b2878-417e-4986-8067-b50a16d16967\") " pod="kube-system/kas-connection-checker-685895c7c9-zdvb2" Oct 30 21:00:14.129336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129188 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-serving-cert\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:14.129336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129211 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1362c712-7332-4a6e-a215-633d2f02acca-serviceca\") pod \"node-ca-cnj4q\" (UID: \"1362c712-7332-4a6e-a215-633d2f02acca\") " pod="openshift-image-registry/node-ca-cnj4q" Oct 30 21:00:14.129336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129237 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4ef3fe7-a7fb-473c-a301-9073d502a86a-data-volume\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:14.129336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129264 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.129336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129274 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e-agent-certs\") pod \"konnectivity-agent-mgsr5\" (UID: \"5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e\") " pod="kube-system/konnectivity-agent-mgsr5" Oct 30 21:00:14.129336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129340 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-serving-cert\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:14.129336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129332 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4ef3fe7-a7fb-473c-a301-9073d502a86a-data-volume\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129375 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-serving-cert\") pod \"console-operator-7cd56b8dbc-c5khs\" (UID: \"1a2fb9ac-a905-4e20-b1be-7928d5e7881a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129395 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ce43826-7550-4c62-8acc-48714541e426-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-frsk7\" (UID: \"4ce43826-7550-4c62-8acc-48714541e426\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129411 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w859\" (UniqueName: \"kubernetes.io/projected/4901add7-3312-4950-ae4c-4d6581380e9e-kube-api-access-4w859\") pod \"ingress-canary-mmzzs\" (UID: \"4901add7-3312-4950-ae4c-4d6581380e9e\") " pod="openshift-ingress-canary/ingress-canary-mmzzs" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129427 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-var-lib-openvswitch\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129443 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxsnl\" (UniqueName: \"kubernetes.io/projected/c0b2deee-2b0c-4c2c-9ac5-debd33f12ba3-kube-api-access-nxsnl\") pod \"kas-connection-checker-685895c7c9-b9fhn\" (UID: \"c0b2deee-2b0c-4c2c-9ac5-debd33f12ba3\") " pod="kube-system/kas-connection-checker-685895c7c9-b9fhn" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129514 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-var-lib-kubelet\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129517 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1362c712-7332-4a6e-a215-633d2f02acca-serviceca\") pod \"node-ca-cnj4q\" (UID: \"1362c712-7332-4a6e-a215-633d2f02acca\") " pod="openshift-image-registry/node-ca-cnj4q" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129557 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4ce43826-7550-4c62-8acc-48714541e426-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-frsk7\" (UID: \"4ce43826-7550-4c62-8acc-48714541e426\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129603 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e20348d-9970-4de7-80be-84f29ef8ece6-tmp\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129622 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03e64eb3-e7df-464d-9382-47cd0e570456-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129644 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hfkg\" (UniqueName: \"kubernetes.io/projected/ef2c01c8-12ef-4294-963e-4e840e2eecff-kube-api-access-9hfkg\") pod \"cluster-samples-operator-6ddb6c8b94-5r7xx\" (UID: \"ef2c01c8-12ef-4294-963e-4e840e2eecff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129664 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-socket-dir-parent\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129682 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e20348d-9970-4de7-80be-84f29ef8ece6-tmp\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129683 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-modprobe-d\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129755 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-sysconfig\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129815 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129835 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-serving-cert\") pod \"console-operator-7cd56b8dbc-c5khs\" (UID: \"1a2fb9ac-a905-4e20-b1be-7928d5e7881a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129862 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwf67\" (UniqueName: \"kubernetes.io/projected/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-kube-api-access-pwf67\") pod \"console-operator-7cd56b8dbc-c5khs\" (UID: \"1a2fb9ac-a905-4e20-b1be-7928d5e7881a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129903 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-host\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129928 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-daemon-config\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129925 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03e64eb3-e7df-464d-9382-47cd0e570456-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129964 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-systemd-units\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.129991 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-slash\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130014 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130092 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-ovn-node-metrics-cert\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130124 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-run-multus-certs\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.130150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130140 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31-tmp-dir\") pod \"node-resolver-54qxj\" (UID: \"a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31\") " pod="openshift-dns/node-resolver-54qxj" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130177 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130196 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2tfx\" (UniqueName: \"kubernetes.io/projected/97872ff8-a1f5-4062-8202-41aa9d8a93bc-kube-api-access-k2tfx\") pod \"iptables-alerter-ww5pw\" (UID: \"97872ff8-a1f5-4062-8202-41aa9d8a93bc\") " pod="openshift-network-operator/iptables-alerter-ww5pw" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130210 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-cni-dir\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130226 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-var-lib-cni-multus\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130244 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-ovnkube-script-lib\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130262 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-daemon-config\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130268 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-kubelet-config\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130299 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31-tmp-dir\") pod \"node-resolver-54qxj\" (UID: \"a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31\") " pod="openshift-dns/node-resolver-54qxj" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130315 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97872ff8-a1f5-4062-8202-41aa9d8a93bc-host-slash\") pod \"iptables-alerter-ww5pw\" (UID: \"97872ff8-a1f5-4062-8202-41aa9d8a93bc\") " pod="openshift-network-operator/iptables-alerter-ww5pw" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130343 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/705e2b0e-10e1-4710-8dfb-bbd862c8c084-config\") pod \"service-ca-operator-544d749c8c-vprbm\" (UID: \"705e2b0e-10e1-4710-8dfb-bbd862c8c084\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130369 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-ovn-node-metrics-cert\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130375 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-etc-openvswitch\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130408 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-log-socket\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130435 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b4ba52c-ad04-4a25-9158-89941367abd3-trusted-ca\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130471 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bc258c0-5d93-4ddb-b49c-9a06167f6185-cni-binary-copy\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130498 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-hostroot\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130486 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-ovnkube-script-lib\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130528 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wbrhm\" (UID: \"9ffe8e19-6888-4eca-ae9d-9d0fc05ba752\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130551 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/705e2b0e-10e1-4710-8dfb-bbd862c8c084-config\") pod \"service-ca-operator-544d749c8c-vprbm\" (UID: \"705e2b0e-10e1-4710-8dfb-bbd862c8c084\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130569 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4ef3fe7-a7fb-473c-a301-9073d502a86a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130604 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-sysctl-conf\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130655 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58ss\" (UniqueName: \"kubernetes.io/projected/1e20348d-9970-4de7-80be-84f29ef8ece6-kube-api-access-d58ss\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130671 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bc258c0-5d93-4ddb-b49c-9a06167f6185-cni-binary-copy\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130685 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-dbus\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130711 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6gqs\" (UniqueName: \"kubernetes.io/projected/705e2b0e-10e1-4710-8dfb-bbd862c8c084-kube-api-access-x6gqs\") pod \"service-ca-operator-544d749c8c-vprbm\" (UID: \"705e2b0e-10e1-4710-8dfb-bbd862c8c084\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130736 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b4ba52c-ad04-4a25-9158-89941367abd3-trusted-ca\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130748 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130777 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ce43826-7550-4c62-8acc-48714541e426-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-frsk7\" (UID: \"4ce43826-7550-4c62-8acc-48714541e426\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130804 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/97872ff8-a1f5-4062-8202-41aa9d8a93bc-iptables-alerter-script\") pod \"iptables-alerter-ww5pw\" (UID: \"97872ff8-a1f5-4062-8202-41aa9d8a93bc\") " pod="openshift-network-operator/iptables-alerter-ww5pw" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130839 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-system-cni-dir\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130871 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-cni-netd\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130889 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-env-overrides\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130910 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03e64eb3-e7df-464d-9382-47cd0e570456-cni-binary-copy\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130944 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e-konnectivity-ca\") pod \"konnectivity-agent-mgsr5\" (UID: \"5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e\") " pod="kube-system/konnectivity-agent-mgsr5" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130979 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-tmp\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130991 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/97872ff8-a1f5-4062-8202-41aa9d8a93bc-iptables-alerter-script\") pod \"iptables-alerter-ww5pw\" (UID: \"97872ff8-a1f5-4062-8202-41aa9d8a93bc\") " pod="openshift-network-operator/iptables-alerter-ww5pw" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.130999 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs\") pod \"network-metrics-daemon-7d6gs\" (UID: \"f6337834-db43-4b8e-a955-9691f72e0f19\") " pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131010 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4ce43826-7550-4c62-8acc-48714541e426-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-frsk7\" (UID: \"4ce43826-7550-4c62-8acc-48714541e426\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131073 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-env-overrides\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131119 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7cv8\" (UniqueName: \"kubernetes.io/projected/f6337834-db43-4b8e-a955-9691f72e0f19-kube-api-access-f7cv8\") pod \"network-metrics-daemon-7d6gs\" (UID: \"f6337834-db43-4b8e-a955-9691f72e0f19\") " pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131137 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/662ead00-b524-4a6e-815b-c3399b3c9e6c-config-volume\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131151 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-etc-selinux\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131156 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e-konnectivity-ca\") pod \"konnectivity-agent-mgsr5\" (UID: \"5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e\") " pod="kube-system/konnectivity-agent-mgsr5" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131167 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131183 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31-hosts-file\") pod \"node-resolver-54qxj\" (UID: \"a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31\") " pod="openshift-dns/node-resolver-54qxj" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131187 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03e64eb3-e7df-464d-9382-47cd0e570456-cni-binary-copy\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131198 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/662ead00-b524-4a6e-815b-c3399b3c9e6c-tmp-dir\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131214 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03e64eb3-e7df-464d-9382-47cd0e570456-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131232 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b61b643-c3cf-453c-875e-25576445ad4f-installation-pull-secrets\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131266 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-tmp\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131305 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5r7xx\" (UID: \"ef2c01c8-12ef-4294-963e-4e840e2eecff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131337 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-default-certificate\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131361 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x799c\" (UniqueName: \"kubernetes.io/projected/8bc258c0-5d93-4ddb-b49c-9a06167f6185-kube-api-access-x799c\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131387 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-trusted-ca\") pod \"console-operator-7cd56b8dbc-c5khs\" (UID: \"1a2fb9ac-a905-4e20-b1be-7928d5e7881a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131390 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03e64eb3-e7df-464d-9382-47cd0e570456-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131431 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-snapshots\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131458 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-etc-kubernetes\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.131444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131481 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-ovnkube-config\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131504 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03e64eb3-e7df-464d-9382-47cd0e570456-system-cni-dir\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131602 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/662ead00-b524-4a6e-815b-c3399b3c9e6c-tmp-dir\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131606 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b61b643-c3cf-453c-875e-25576445ad4f-installation-pull-secrets\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131657 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g67v\" (UniqueName: \"kubernetes.io/projected/93f2c084-a1d2-4bda-8a41-c01829cc81ba-kube-api-access-6g67v\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131681 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-ovnkube-config\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131678 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-snapshots\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131684 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhhrx\" (UniqueName: \"kubernetes.io/projected/1362c712-7332-4a6e-a215-633d2f02acca-kube-api-access-dhhrx\") pod \"node-ca-cnj4q\" (UID: \"1362c712-7332-4a6e-a215-633d2f02acca\") " pod="openshift-image-registry/node-ca-cnj4q" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131738 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/662ead00-b524-4a6e-815b-c3399b3c9e6c-config-volume\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131750 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131761 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-default-certificate\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131845 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wbrhm\" (UID: \"9ffe8e19-6888-4eca-ae9d-9d0fc05ba752\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131887 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-cni-bin\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131910 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-trusted-ca\") pod \"console-operator-7cd56b8dbc-c5khs\" (UID: \"1a2fb9ac-a905-4e20-b1be-7928d5e7881a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131968 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b4ba52c-ad04-4a25-9158-89941367abd3-installation-pull-secrets\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131995 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-5r7xx\" (UID: \"ef2c01c8-12ef-4294-963e-4e840e2eecff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.131999 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2749\" (UniqueName: \"kubernetes.io/projected/0b77d4c7-37eb-4bad-8367-31346289726e-kube-api-access-l2749\") pod \"cluster-monitoring-operator-884576c57-v5rrz\" (UID: \"0b77d4c7-37eb-4bad-8367-31346289726e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132061 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b61b643-c3cf-453c-875e-25576445ad4f-trusted-ca\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132077 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4901add7-3312-4950-ae4c-4d6581380e9e-cert\") pod \"ingress-canary-mmzzs\" (UID: \"4901add7-3312-4950-ae4c-4d6581380e9e\") " pod="openshift-ingress-canary/ingress-canary-mmzzs" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132079 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-nginx-conf\") pod \"networking-console-plugin-86c478bb88-wbrhm\" (UID: \"9ffe8e19-6888-4eca-ae9d-9d0fc05ba752\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132091 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03e64eb3-e7df-464d-9382-47cd0e570456-cnibin\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132108 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v5rrz\" (UID: \"0b77d4c7-37eb-4bad-8367-31346289726e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132127 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b61b643-c3cf-453c-875e-25576445ad4f-registry-certificates\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132172 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhvqf\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-kube-api-access-rhvqf\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132188 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b4ba52c-ad04-4a25-9158-89941367abd3-installation-pull-secrets\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132209 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrt6b\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-kube-api-access-vrt6b\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132238 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tf9r\" (UniqueName: \"kubernetes.io/projected/662ead00-b524-4a6e-815b-c3399b3c9e6c-kube-api-access-6tf9r\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132269 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qzp2\" (UniqueName: \"kubernetes.io/projected/c790e080-b546-409c-8ed2-1fc23982984b-kube-api-access-6qzp2\") pod \"migrator-66df5d496b-5v597\" (UID: \"c790e080-b546-409c-8ed2-1fc23982984b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5v597" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132287 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4901add7-3312-4950-ae4c-4d6581380e9e-cert\") pod \"ingress-canary-mmzzs\" (UID: \"4901add7-3312-4950-ae4c-4d6581380e9e\") " pod="openshift-ingress-canary/ingress-canary-mmzzs" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132291 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b61b643-c3cf-453c-875e-25576445ad4f-registry-certificates\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132296 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132331 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-stats-auth\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132359 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-cnibin\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132434 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67blk\" (UniqueName: \"kubernetes.io/projected/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-kube-api-access-67blk\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132477 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4ef3fe7-a7fb-473c-a301-9073d502a86a-crio-socket\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132507 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-socket-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132510 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-stats-auth\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132532 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-sys-fs\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132563 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b61b643-c3cf-453c-875e-25576445ad4f-image-registry-private-configuration\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132602 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b61b643-c3cf-453c-875e-25576445ad4f-ca-trust-extracted\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132646 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b61b643-c3cf-453c-875e-25576445ad4f-trusted-ca\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132690 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132728 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132738 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b61b643-c3cf-453c-875e-25576445ad4f-image-registry-private-configuration\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132728 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b61b643-c3cf-453c-875e-25576445ad4f-ca-trust-extracted\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132758 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-conf-dir\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132784 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xcm4\" (UniqueName: \"kubernetes.io/projected/03e64eb3-e7df-464d-9382-47cd0e570456-kube-api-access-9xcm4\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132801 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-registration-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.132829 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmcw2\" (UniqueName: \"kubernetes.io/projected/6d9c7935-1d3d-4323-ac72-008a394e7f31-kube-api-access-nmcw2\") pod \"kas-connection-checker-685895c7c9-zr75n\" (UID: \"6d9c7935-1d3d-4323-ac72-008a394e7f31\") " pod="kube-system/kas-connection-checker-685895c7c9-zr75n" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.133023 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b4ba52c-ad04-4a25-9158-89941367abd3-image-registry-private-configuration\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.133060 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b67sr\" (UniqueName: \"kubernetes.io/projected/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-kube-api-access-b67sr\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.133077 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-certificates\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.133091 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-bound-sa-token\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.133173 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b4ba52c-ad04-4a25-9158-89941367abd3-image-registry-private-configuration\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.133212 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2mms\" (UniqueName: \"kubernetes.io/projected/a66e14f7-3c72-48a5-88e8-e9f78a7c7ff5-kube-api-access-h2mms\") pod \"volume-data-source-validator-58c9f5f46d-87c42\" (UID: \"a66e14f7-3c72-48a5-88e8-e9f78a7c7ff5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87c42" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.133231 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-certificates\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.133246 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9bws\" (UniqueName: \"kubernetes.io/projected/b4ef3fe7-a7fb-473c-a301-9073d502a86a-kube-api-access-r9bws\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:14.133728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.133278 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-run\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.158235 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.158214 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxsnl\" (UniqueName: \"kubernetes.io/projected/c0b2deee-2b0c-4c2c-9ac5-debd33f12ba3-kube-api-access-nxsnl\") pod \"kas-connection-checker-685895c7c9-b9fhn\" (UID: \"c0b2deee-2b0c-4c2c-9ac5-debd33f12ba3\") " pod="kube-system/kas-connection-checker-685895c7c9-b9fhn" Oct 30 21:00:14.158378 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.158363 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgg8n\" (UniqueName: \"kubernetes.io/projected/3358db14-a01d-479e-a742-bdcd1ca80527-kube-api-access-cgg8n\") pod \"network-check-source-6f78d59f99-j5tcr\" (UID: \"3358db14-a01d-479e-a742-bdcd1ca80527\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-j5tcr" Oct 30 21:00:14.166387 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.166364 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2tfx\" (UniqueName: \"kubernetes.io/projected/97872ff8-a1f5-4062-8202-41aa9d8a93bc-kube-api-access-k2tfx\") pod \"iptables-alerter-ww5pw\" (UID: \"97872ff8-a1f5-4062-8202-41aa9d8a93bc\") " pod="openshift-network-operator/iptables-alerter-ww5pw" Oct 30 21:00:14.166490 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.166464 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b2ss\" (UniqueName: \"kubernetes.io/projected/a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31-kube-api-access-9b2ss\") pod \"node-resolver-54qxj\" (UID: \"a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31\") " pod="openshift-dns/node-resolver-54qxj" Oct 30 21:00:14.166490 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.166480 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2k65\" (UniqueName: \"kubernetes.io/projected/0f0b2878-417e-4986-8067-b50a16d16967-kube-api-access-z2k65\") pod \"kas-connection-checker-685895c7c9-zdvb2\" (UID: \"0f0b2878-417e-4986-8067-b50a16d16967\") " pod="kube-system/kas-connection-checker-685895c7c9-zdvb2" Oct 30 21:00:14.166613 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.166511 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvl2l\" (UniqueName: \"kubernetes.io/projected/4ce43826-7550-4c62-8acc-48714541e426-kube-api-access-gvl2l\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-frsk7\" (UID: \"4ce43826-7550-4c62-8acc-48714541e426\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7" Oct 30 21:00:14.166678 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.166662 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58ss\" (UniqueName: \"kubernetes.io/projected/1e20348d-9970-4de7-80be-84f29ef8ece6-kube-api-access-d58ss\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.166777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.166760 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w859\" (UniqueName: \"kubernetes.io/projected/4901add7-3312-4950-ae4c-4d6581380e9e-kube-api-access-4w859\") pod \"ingress-canary-mmzzs\" (UID: \"4901add7-3312-4950-ae4c-4d6581380e9e\") " pod="openshift-ingress-canary/ingress-canary-mmzzs" Oct 30 21:00:14.166847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.166829 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz6tt\" (UniqueName: \"kubernetes.io/projected/4e46662c-bfe6-45f8-8a23-89ea5fc92ae5-kube-api-access-dz6tt\") pod \"network-check-target-mfgjb\" (UID: \"4e46662c-bfe6-45f8-8a23-89ea5fc92ae5\") " pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 21:00:14.172806 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.172791 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7cv8\" (UniqueName: \"kubernetes.io/projected/f6337834-db43-4b8e-a955-9691f72e0f19-kube-api-access-f7cv8\") pod \"network-metrics-daemon-7d6gs\" (UID: \"f6337834-db43-4b8e-a955-9691f72e0f19\") " pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 21:00:14.176926 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.176902 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz752\" (UniqueName: \"kubernetes.io/projected/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-kube-api-access-cz752\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:14.177884 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.177869 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6gqs\" (UniqueName: \"kubernetes.io/projected/705e2b0e-10e1-4710-8dfb-bbd862c8c084-kube-api-access-x6gqs\") pod \"service-ca-operator-544d749c8c-vprbm\" (UID: \"705e2b0e-10e1-4710-8dfb-bbd862c8c084\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm" Oct 30 21:00:14.178607 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.178587 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrt6b\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-kube-api-access-vrt6b\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:14.179429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.179411 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x799c\" (UniqueName: \"kubernetes.io/projected/8bc258c0-5d93-4ddb-b49c-9a06167f6185-kube-api-access-x799c\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.179630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.179615 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g67v\" (UniqueName: \"kubernetes.io/projected/93f2c084-a1d2-4bda-8a41-c01829cc81ba-kube-api-access-6g67v\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 21:00:14.179742 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.179727 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwf67\" (UniqueName: \"kubernetes.io/projected/1a2fb9ac-a905-4e20-b1be-7928d5e7881a-kube-api-access-pwf67\") pod \"console-operator-7cd56b8dbc-c5khs\" (UID: \"1a2fb9ac-a905-4e20-b1be-7928d5e7881a\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:14.180890 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.180874 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qzp2\" (UniqueName: \"kubernetes.io/projected/c790e080-b546-409c-8ed2-1fc23982984b-kube-api-access-6qzp2\") pod \"migrator-66df5d496b-5v597\" (UID: \"c790e080-b546-409c-8ed2-1fc23982984b\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-5v597" Oct 30 21:00:14.181041 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.181016 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2749\" (UniqueName: \"kubernetes.io/projected/0b77d4c7-37eb-4bad-8367-31346289726e-kube-api-access-l2749\") pod \"cluster-monitoring-operator-884576c57-v5rrz\" (UID: \"0b77d4c7-37eb-4bad-8367-31346289726e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:14.181935 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.181918 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xcm4\" (UniqueName: \"kubernetes.io/projected/03e64eb3-e7df-464d-9382-47cd0e570456-kube-api-access-9xcm4\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 21:00:14.182200 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.182184 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhhrx\" (UniqueName: \"kubernetes.io/projected/1362c712-7332-4a6e-a215-633d2f02acca-kube-api-access-dhhrx\") pod \"node-ca-cnj4q\" (UID: \"1362c712-7332-4a6e-a215-633d2f02acca\") " pod="openshift-image-registry/node-ca-cnj4q" Oct 30 21:00:14.182790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.182775 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhvqf\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-kube-api-access-rhvqf\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:14.182825 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.182788 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9bws\" (UniqueName: \"kubernetes.io/projected/b4ef3fe7-a7fb-473c-a301-9073d502a86a-kube-api-access-r9bws\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:14.183559 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.183543 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2mms\" (UniqueName: \"kubernetes.io/projected/a66e14f7-3c72-48a5-88e8-e9f78a7c7ff5-kube-api-access-h2mms\") pod \"volume-data-source-validator-58c9f5f46d-87c42\" (UID: \"a66e14f7-3c72-48a5-88e8-e9f78a7c7ff5\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-87c42" Oct 30 21:00:14.183824 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.183809 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tf9r\" (UniqueName: \"kubernetes.io/projected/662ead00-b524-4a6e-815b-c3399b3c9e6c-kube-api-access-6tf9r\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:14.184403 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.184011 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-bound-sa-token\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:14.185203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.185187 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hfkg\" (UniqueName: \"kubernetes.io/projected/ef2c01c8-12ef-4294-963e-4e840e2eecff-kube-api-access-9hfkg\") pod \"cluster-samples-operator-6ddb6c8b94-5r7xx\" (UID: \"ef2c01c8-12ef-4294-963e-4e840e2eecff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" Oct 30 21:00:14.185662 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.185644 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67blk\" (UniqueName: \"kubernetes.io/projected/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-kube-api-access-67blk\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.186243 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.186225 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-bound-sa-token\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:14.186788 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.186773 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b67sr\" (UniqueName: \"kubernetes.io/projected/ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6-kube-api-access-b67sr\") pod \"insights-operator-7c4986d449-m7m4c\" (UID: \"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6\") " pod="openshift-insights/insights-operator-7c4986d449-m7m4c" Oct 30 21:00:14.188903 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.188887 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmcw2\" (UniqueName: \"kubernetes.io/projected/6d9c7935-1d3d-4323-ac72-008a394e7f31-kube-api-access-nmcw2\") pod \"kas-connection-checker-685895c7c9-zr75n\" (UID: \"6d9c7935-1d3d-4323-ac72-008a394e7f31\") " pod="kube-system/kas-connection-checker-685895c7c9-zr75n" Oct 30 21:00:14.234454 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234427 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-lib-modules\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.234454 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234456 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.234640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234473 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-var-lib-openvswitch\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.234640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234490 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-var-lib-kubelet\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.234640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234506 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-socket-dir-parent\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.234640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234520 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-modprobe-d\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.234640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234534 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-sysconfig\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.234640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234540 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-var-lib-openvswitch\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.234640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234555 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 21:00:14.234640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234538 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.234640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234590 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-host\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.234640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234597 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-var-lib-kubelet\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.234640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234617 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-systemd-units\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.234640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234626 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-socket-dir-parent\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.234640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234604 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 21:00:14.234640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234640 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-sysconfig\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234645 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-lib-modules\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234655 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-systemd-units\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234642 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-host\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234636 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-modprobe-d\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234685 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-slash\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234709 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234725 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-run-multus-certs\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234735 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-slash\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234740 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234742 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234764 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-run-multus-certs\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234777 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-cni-dir\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234801 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-var-lib-cni-multus\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234817 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-kubelet-config\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.234819 4926 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234831 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97872ff8-a1f5-4062-8202-41aa9d8a93bc-host-slash\") pod \"iptables-alerter-ww5pw\" (UID: \"97872ff8-a1f5-4062-8202-41aa9d8a93bc\") " pod="openshift-network-operator/iptables-alerter-ww5pw" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234853 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-var-lib-cni-multus\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.234882 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls podName:b4ef3fe7-a7fb-473c-a301-9073d502a86a nodeName:}" failed. No retries permitted until 2026-10-30 21:00:14.734866962 +0000 UTC m=+1.855940710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q9bnp" (UID: "b4ef3fe7-a7fb-473c-a301-9073d502a86a") : secret "insights-runtime-extractor-tls" not found Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234884 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-kubelet-config\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234885 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/97872ff8-a1f5-4062-8202-41aa9d8a93bc-host-slash\") pod \"iptables-alerter-ww5pw\" (UID: \"97872ff8-a1f5-4062-8202-41aa9d8a93bc\") " pod="openshift-network-operator/iptables-alerter-ww5pw" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234853 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-etc-openvswitch\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234934 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-log-socket\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234951 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-hostroot\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234966 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wbrhm\" (UID: \"9ffe8e19-6888-4eca-ae9d-9d0fc05ba752\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234989 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4ef3fe7-a7fb-473c-a301-9073d502a86a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235005 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-log-socket\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234906 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-cni-dir\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.234916 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-etc-openvswitch\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235057 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-hostroot\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235068 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-sysctl-conf\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.235081 4926 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235094 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-dbus\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235126 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:14.235116 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.235133 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert podName:9ffe8e19-6888-4eca-ae9d-9d0fc05ba752 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:14.735118004 +0000 UTC m=+1.856191745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wbrhm" (UID: "9ffe8e19-6888-4eca-ae9d-9d0fc05ba752") : secret "networking-console-plugin-cert" not found Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235159 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-system-cni-dir\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235160 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e-dbus\") pod \"global-pull-secret-syncer-m9dm5\" (UID: \"5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e\") " pod="kube-system/global-pull-secret-syncer-m9dm5" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235181 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-cni-netd\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235209 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs\") pod \"network-metrics-daemon-7d6gs\" (UID: \"f6337834-db43-4b8e-a955-9691f72e0f19\") " pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235214 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-cni-netd\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235221 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-sysctl-conf\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.235187 4926 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235211 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-system-cni-dir\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.235240 4926 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56886949f8-xwhft: secret "image-registry-tls" not found Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235234 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-etc-selinux\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.235280 4926 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235340 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31-hosts-file\") pod \"node-resolver-54qxj\" (UID: \"a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31\") " pod="openshift-dns/node-resolver-54qxj" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235346 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-etc-selinux\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.235355 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls podName:9b61b643-c3cf-453c-875e-25576445ad4f nodeName:}" failed. No retries permitted until 2026-10-30 21:00:14.735344807 +0000 UTC m=+1.856418555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls") pod "image-registry-56886949f8-xwhft" (UID: "9b61b643-c3cf-453c-875e-25576445ad4f") : secret "image-registry-tls" not found Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.235376 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs podName:f6337834-db43-4b8e-a955-9691f72e0f19 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:14.735366797 +0000 UTC m=+1.856440550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs") pod "network-metrics-daemon-7d6gs" (UID: "f6337834-db43-4b8e-a955-9691f72e0f19") : secret "metrics-daemon-secret" not found Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235377 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31-hosts-file\") pod \"node-resolver-54qxj\" (UID: \"a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31\") " pod="openshift-dns/node-resolver-54qxj" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235407 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-etc-kubernetes\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235428 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03e64eb3-e7df-464d-9382-47cd0e570456-system-cni-dir\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235444 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-cni-bin\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235465 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03e64eb3-e7df-464d-9382-47cd0e570456-cnibin\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235473 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-etc-kubernetes\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235489 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03e64eb3-e7df-464d-9382-47cd0e570456-system-cni-dir\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235498 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-cni-bin\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235504 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03e64eb3-e7df-464d-9382-47cd0e570456-cnibin\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235490 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v5rrz\" (UID: \"0b77d4c7-37eb-4bad-8367-31346289726e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235540 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.235549 4926 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235564 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-cnibin\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.235589 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls podName:0b77d4c7-37eb-4bad-8367-31346289726e nodeName:}" failed. No retries permitted until 2026-10-30 21:00:14.735577567 +0000 UTC m=+1.856651308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-v5rrz" (UID: "0b77d4c7-37eb-4bad-8367-31346289726e") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235605 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4ef3fe7-a7fb-473c-a301-9073d502a86a-crio-socket\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235615 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-cnibin\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235629 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-socket-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235633 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b4ef3fe7-a7fb-473c-a301-9073d502a86a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235654 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-sys-fs\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235659 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b4ef3fe7-a7fb-473c-a301-9073d502a86a-crio-socket\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.235663 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle podName:276d4076-cd5a-45d6-b4f8-2de14f9c9a0d nodeName:}" failed. No retries permitted until 2026-10-30 21:00:14.735654141 +0000 UTC m=+1.856727884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle") pod "router-default-6789dbc4b5-5gqgl" (UID: "276d4076-cd5a-45d6-b4f8-2de14f9c9a0d") : configmap references non-existent config key: service-ca.crt Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235709 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-socket-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235695 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235710 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-sys-fs\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235748 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235779 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-conf-dir\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.235763 4926 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235802 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-registration-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.235789 4926 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235823 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-run\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235839 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-run-k8s-cni-cncf-io\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.235848 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls podName:662ead00-b524-4a6e-815b-c3399b3c9e6c nodeName:}" failed. No retries permitted until 2026-10-30 21:00:14.735836226 +0000 UTC m=+1.856909950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls") pod "dns-default-888xx" (UID: "662ead00-b524-4a6e-815b-c3399b3c9e6c") : secret "dns-default-metrics-tls" not found Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235843 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-multus-conf-dir\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235863 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-run-k8s-cni-cncf-io\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235872 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-run\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.235881 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs podName:276d4076-cd5a-45d6-b4f8-2de14f9c9a0d nodeName:}" failed. No retries permitted until 2026-10-30 21:00:14.735875678 +0000 UTC m=+1.856949401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs") pod "router-default-6789dbc4b5-5gqgl" (UID: "276d4076-cd5a-45d6-b4f8-2de14f9c9a0d") : secret "router-metrics-certs-default" not found Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235894 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-run-netns\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235900 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-registration-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235915 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-run-openvswitch\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235931 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-run-netns\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235934 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-node-log\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235951 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-node-log\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235955 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-var-lib-cni-bin\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235970 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1362c712-7332-4a6e-a215-633d2f02acca-host\") pod \"node-ca-cnj4q\" (UID: \"1362c712-7332-4a6e-a215-633d2f02acca\") " pod="openshift-image-registry/node-ca-cnj4q" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235982 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-run-openvswitch\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.235985 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-sysctl-d\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236012 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1362c712-7332-4a6e-a215-633d2f02acca-host\") pod \"node-ca-cnj4q\" (UID: \"1362c712-7332-4a6e-a215-633d2f02acca\") " pod="openshift-image-registry/node-ca-cnj4q" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236012 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-device-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236062 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/93f2c084-a1d2-4bda-8a41-c01829cc81ba-device-dir\") pod \"aws-ebs-csi-driver-node-vgk87\" (UID: \"93f2c084-a1d2-4bda-8a41-c01829cc81ba\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vgk87" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236071 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-run-ovn\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236010 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-var-lib-cni-bin\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236079 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-sysctl-d\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236098 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5r7xx\" (UID: \"ef2c01c8-12ef-4294-963e-4e840e2eecff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236115 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-kubelet\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236131 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-kubernetes\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236159 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-host-kubelet\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236171 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-kubernetes\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236179 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236196 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-run-ovn\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236198 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-run-systemd\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236215 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/88f212cc-4eb8-4a49-9c13-a1d395ab43c6-run-systemd\") pod \"ovnkube-node-vtgj9\" (UID: \"88f212cc-4eb8-4a49-9c13-a1d395ab43c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236227 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-systemd\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236248 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-sys\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236271 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03e64eb3-e7df-464d-9382-47cd0e570456-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.236275 4926 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236296 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-run-netns\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.236301 4926 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758fcd4795-j6bts: secret "image-registry-tls" not found Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236311 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-etc-systemd\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236337 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03e64eb3-e7df-464d-9382-47cd0e570456-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9gv9s\" (UID: \"03e64eb3-e7df-464d-9382-47cd0e570456\") " pod="openshift-multus/multus-additional-cni-plugins-9gv9s" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236346 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1e20348d-9970-4de7-80be-84f29ef8ece6-sys\") pod \"tuned-rkcv2\" (UID: \"1e20348d-9970-4de7-80be-84f29ef8ece6\") " pod="openshift-cluster-node-tuning-operator/tuned-rkcv2" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.236358 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls podName:6b4ba52c-ad04-4a25-9158-89941367abd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:14.736342101 +0000 UTC m=+1.857415827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls") pod "image-registry-758fcd4795-j6bts" (UID: "6b4ba52c-ad04-4a25-9158-89941367abd3") : secret "image-registry-tls" not found Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236340 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-run-netns\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236378 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-var-lib-kubelet\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236391 4926 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:00:14.236351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.236419 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bc258c0-5d93-4ddb-b49c-9a06167f6185-host-var-lib-kubelet\") pod \"multus-ggbwl\" (UID: \"8bc258c0-5d93-4ddb-b49c-9a06167f6185\") " pod="openshift-multus/multus-ggbwl" Oct 30 21:00:14.238769 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.238459 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef2c01c8-12ef-4294-963e-4e840e2eecff-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-5r7xx\" (UID: \"ef2c01c8-12ef-4294-963e-4e840e2eecff\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" Oct 30 21:00:14.320378 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.320302 4926 scope.go:122] "RemoveContainer" containerID="cb966d4ceaaab215d2d47172a0b5244daddd96c1007b782eaaa1ed2d0d47b7ee" Oct 30 21:00:14.321348 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.321328 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" Oct 30 21:00:14.400067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.399865 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-54qxj_a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31/dns-node-resolver/0.log" Oct 30 21:00:14.633400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.633323 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx"] Oct 30 21:00:14.709172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.709155 4926 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:00:14.741977 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.741948 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wbrhm\" (UID: \"9ffe8e19-6888-4eca-ae9d-9d0fc05ba752\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" Oct 30 21:00:14.742108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.741986 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:14.742108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.742007 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs\") pod \"network-metrics-daemon-7d6gs\" (UID: \"f6337834-db43-4b8e-a955-9691f72e0f19\") " pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 21:00:14.742108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.742061 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v5rrz\" (UID: \"0b77d4c7-37eb-4bad-8367-31346289726e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:14.742108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.742082 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:14.742108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.742100 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:14.742314 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.742114 4926 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:00:14.742314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.742122 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:14.742314 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.742119 4926 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:00:14.742314 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.742169 4926 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:00:14.742314 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.742181 4926 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56886949f8-xwhft: secret "image-registry-tls" not found Oct 30 21:00:14.742314 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.742190 4926 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 21:00:14.742314 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.742196 4926 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:00:14.742314 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.742184 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert podName:9ffe8e19-6888-4eca-ae9d-9d0fc05ba752 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:15.742163485 +0000 UTC m=+2.863237227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wbrhm" (UID: "9ffe8e19-6888-4eca-ae9d-9d0fc05ba752") : secret "networking-console-plugin-cert" not found Oct 30 21:00:14.742314 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.742252 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle podName:276d4076-cd5a-45d6-b4f8-2de14f9c9a0d nodeName:}" failed. No retries permitted until 2026-10-30 21:00:15.742238319 +0000 UTC m=+2.863312057 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle") pod "router-default-6789dbc4b5-5gqgl" (UID: "276d4076-cd5a-45d6-b4f8-2de14f9c9a0d") : configmap references non-existent config key: service-ca.crt Oct 30 21:00:14.742314 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.742254 4926 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 21:00:14.742314 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.742268 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls podName:662ead00-b524-4a6e-815b-c3399b3c9e6c nodeName:}" failed. No retries permitted until 2026-10-30 21:00:15.742260739 +0000 UTC m=+2.863334462 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls") pod "dns-default-888xx" (UID: "662ead00-b524-4a6e-815b-c3399b3c9e6c") : secret "dns-default-metrics-tls" not found Oct 30 21:00:14.742314 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.742294 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs podName:276d4076-cd5a-45d6-b4f8-2de14f9c9a0d nodeName:}" failed. No retries permitted until 2026-10-30 21:00:15.742280052 +0000 UTC m=+2.863353790 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs") pod "router-default-6789dbc4b5-5gqgl" (UID: "276d4076-cd5a-45d6-b4f8-2de14f9c9a0d") : secret "router-metrics-certs-default" not found Oct 30 21:00:14.742314 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.742319 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls podName:9b61b643-c3cf-453c-875e-25576445ad4f nodeName:}" failed. No retries permitted until 2026-10-30 21:00:15.742310321 +0000 UTC m=+2.863384055 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls") pod "image-registry-56886949f8-xwhft" (UID: "9b61b643-c3cf-453c-875e-25576445ad4f") : secret "image-registry-tls" not found Oct 30 21:00:14.742692 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.742337 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs podName:f6337834-db43-4b8e-a955-9691f72e0f19 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:15.742328133 +0000 UTC m=+2.863401867 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs") pod "network-metrics-daemon-7d6gs" (UID: "f6337834-db43-4b8e-a955-9691f72e0f19") : secret "metrics-daemon-secret" not found Oct 30 21:00:14.742692 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.742357 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls podName:0b77d4c7-37eb-4bad-8367-31346289726e nodeName:}" failed. No retries permitted until 2026-10-30 21:00:15.742348649 +0000 UTC m=+2.863422384 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-v5rrz" (UID: "0b77d4c7-37eb-4bad-8367-31346289726e") : secret "cluster-monitoring-operator-tls" not found Oct 30 21:00:14.742692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.742419 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:14.742692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:14.742487 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:14.742692 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.742519 4926 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:00:14.742692 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.742529 4926 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758fcd4795-j6bts: secret "image-registry-tls" not found Oct 30 21:00:14.742692 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.742551 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls podName:6b4ba52c-ad04-4a25-9158-89941367abd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:15.742544754 +0000 UTC m=+2.863618520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls") pod "image-registry-758fcd4795-j6bts" (UID: "6b4ba52c-ad04-4a25-9158-89941367abd3") : secret "image-registry-tls" not found Oct 30 21:00:14.742692 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.742586 4926 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:00:14.742692 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:14.742622 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls podName:b4ef3fe7-a7fb-473c-a301-9073d502a86a nodeName:}" failed. No retries permitted until 2026-10-30 21:00:15.74261142 +0000 UTC m=+2.863685157 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q9bnp" (UID: "b4ef3fe7-a7fb-473c-a301-9073d502a86a") : secret "insights-runtime-extractor-tls" not found Oct 30 21:00:15.068227 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:15.068191 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" event={"ID":"ef2c01c8-12ef-4294-963e-4e840e2eecff","Type":"ContainerStarted","Data":"547568088b8f1017ecee735dcbedcf4fa451a3c6fd0bcf369613e4fe4190f81c"} Oct 30 21:00:15.069991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:15.069972 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:00:15.070129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:15.070023 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" event={"ID":"1a2fb9ac-a905-4e20-b1be-7928d5e7881a","Type":"ContainerStarted","Data":"097d514519d15db5230b7a31d4dea8e11db1e76361270d3fb5a17c33d738c478"} Oct 30 21:00:15.395927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:15.395858 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cnj4q_1362c712-7332-4a6e-a215-633d2f02acca/node-ca/0.log" Oct 30 21:00:15.752130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:15.752099 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs\") pod \"network-metrics-daemon-7d6gs\" (UID: \"f6337834-db43-4b8e-a955-9691f72e0f19\") " pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 21:00:15.752281 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:15.752145 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v5rrz\" (UID: \"0b77d4c7-37eb-4bad-8367-31346289726e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:15.752281 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:15.752168 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:15.752281 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:15.752215 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:15.752281 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:15.752238 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:15.752416 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:15.752312 4926 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:00:15.752416 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:15.752342 4926 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:00:15.752416 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:15.752383 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs podName:f6337834-db43-4b8e-a955-9691f72e0f19 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:17.752365126 +0000 UTC m=+4.873438862 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs") pod "network-metrics-daemon-7d6gs" (UID: "f6337834-db43-4b8e-a955-9691f72e0f19") : secret "metrics-daemon-secret" not found Oct 30 21:00:15.752416 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:15.752404 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls podName:662ead00-b524-4a6e-815b-c3399b3c9e6c nodeName:}" failed. No retries permitted until 2026-10-30 21:00:17.752395274 +0000 UTC m=+4.873468997 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls") pod "dns-default-888xx" (UID: "662ead00-b524-4a6e-815b-c3399b3c9e6c") : secret "dns-default-metrics-tls" not found Oct 30 21:00:15.752543 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:15.752426 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:15.752543 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:15.752509 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:15.752543 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:15.752519 4926 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:00:15.752543 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:15.752537 4926 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758fcd4795-j6bts: secret "image-registry-tls" not found Oct 30 21:00:15.752707 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:15.752536 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle podName:276d4076-cd5a-45d6-b4f8-2de14f9c9a0d nodeName:}" failed. No retries permitted until 2026-10-30 21:00:17.75251783 +0000 UTC m=+4.873591558 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle") pod "router-default-6789dbc4b5-5gqgl" (UID: "276d4076-cd5a-45d6-b4f8-2de14f9c9a0d") : configmap references non-existent config key: service-ca.crt Oct 30 21:00:15.752707 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:15.752575 4926 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:00:15.752707 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:15.752603 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls podName:6b4ba52c-ad04-4a25-9158-89941367abd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:17.752582283 +0000 UTC m=+4.873656018 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls") pod "image-registry-758fcd4795-j6bts" (UID: "6b4ba52c-ad04-4a25-9158-89941367abd3") : secret "image-registry-tls" not found Oct 30 21:00:15.752707 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:15.752651 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls podName:b4ef3fe7-a7fb-473c-a301-9073d502a86a nodeName:}" failed. No retries permitted until 2026-10-30 21:00:17.75264049 +0000 UTC m=+4.873714232 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q9bnp" (UID: "b4ef3fe7-a7fb-473c-a301-9073d502a86a") : secret "insights-runtime-extractor-tls" not found Oct 30 21:00:15.752707 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:15.752668 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wbrhm\" (UID: \"9ffe8e19-6888-4eca-ae9d-9d0fc05ba752\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" Oct 30 21:00:15.752924 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:15.752721 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:15.752924 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:15.752796 4926 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:00:15.752924 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:15.752812 4926 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:00:15.752924 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:15.752821 4926 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56886949f8-xwhft: secret "image-registry-tls" not found Oct 30 21:00:15.752924 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:15.752845 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert podName:9ffe8e19-6888-4eca-ae9d-9d0fc05ba752 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:17.752831532 +0000 UTC m=+4.873905269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wbrhm" (UID: "9ffe8e19-6888-4eca-ae9d-9d0fc05ba752") : secret "networking-console-plugin-cert" not found Oct 30 21:00:15.752924 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:15.752862 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls podName:9b61b643-c3cf-453c-875e-25576445ad4f nodeName:}" failed. No retries permitted until 2026-10-30 21:00:17.752854143 +0000 UTC m=+4.873927874 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls") pod "image-registry-56886949f8-xwhft" (UID: "9b61b643-c3cf-453c-875e-25576445ad4f") : secret "image-registry-tls" not found Oct 30 21:00:15.754603 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:15.754577 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0b77d4c7-37eb-4bad-8367-31346289726e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-v5rrz\" (UID: \"0b77d4c7-37eb-4bad-8367-31346289726e\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:15.754697 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:15.754610 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-metrics-certs\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:15.821698 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:15.821670 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" Oct 30 21:00:16.127363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:16.127236 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz"] Oct 30 21:00:16.131749 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:16.131702 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b77d4c7_37eb_4bad_8367_31346289726e.slice/crio-4cf19f7e7ef80caf0387d61c2e6247c533d2d48adb4bca604f3243b08f32cfec WatchSource:0}: Error finding container 4cf19f7e7ef80caf0387d61c2e6247c533d2d48adb4bca604f3243b08f32cfec: Status 404 returned error can't find the container with id 4cf19f7e7ef80caf0387d61c2e6247c533d2d48adb4bca604f3243b08f32cfec Oct 30 21:00:16.397023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:16.396938 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mmzzs_4901add7-3312-4950-ae4c-4d6581380e9e/serve-healthcheck-canary/0.log" Oct 30 21:00:16.595940 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:16.595856 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-5v597_c790e080-b546-409c-8ed2-1fc23982984b/migrator/0.log" Oct 30 21:00:16.878747 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:16.878723 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 21:00:16.997106 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:16.997066 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-frsk7_4ce43826-7550-4c62-8acc-48714541e426/kube-storage-version-migrator-operator/0.log" Oct 30 21:00:17.083815 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:17.083723 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" event={"ID":"ef2c01c8-12ef-4294-963e-4e840e2eecff","Type":"ContainerStarted","Data":"bcbf401a952878730d09dba49fd4801f0255be33281865d1001698703a226b64"} Oct 30 21:00:17.083815 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:17.083801 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-5r7xx" event={"ID":"ef2c01c8-12ef-4294-963e-4e840e2eecff","Type":"ContainerStarted","Data":"3ab7e4954be90e42d7988d4ebc91e3889e6943cb632af20a4e7312e0ef50ae37"} Oct 30 21:00:17.085082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:17.085059 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" event={"ID":"0b77d4c7-37eb-4bad-8367-31346289726e","Type":"ContainerStarted","Data":"4cf19f7e7ef80caf0387d61c2e6247c533d2d48adb4bca604f3243b08f32cfec"} Oct 30 21:00:17.773385 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:17.773345 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:17.773887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:17.773408 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:17.773887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:17.773466 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:17.773887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:17.773503 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wbrhm\" (UID: \"9ffe8e19-6888-4eca-ae9d-9d0fc05ba752\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" Oct 30 21:00:17.773887 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:17.773523 4926 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:00:17.773887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:17.773534 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:17.773887 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:17.773540 4926 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-758fcd4795-j6bts: secret "image-registry-tls" not found Oct 30 21:00:17.773887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:17.773562 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs\") pod \"network-metrics-daemon-7d6gs\" (UID: \"f6337834-db43-4b8e-a955-9691f72e0f19\") " pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 21:00:17.773887 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:17.773599 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls podName:6b4ba52c-ad04-4a25-9158-89941367abd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:21.773581571 +0000 UTC m=+8.894655318 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls") pod "image-registry-758fcd4795-j6bts" (UID: "6b4ba52c-ad04-4a25-9158-89941367abd3") : secret "image-registry-tls" not found Oct 30 21:00:17.773887 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:17.773621 4926 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:00:17.773887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:17.773636 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:17.773887 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:17.773676 4926 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 21:00:17.773887 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:17.773680 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls podName:b4ef3fe7-a7fb-473c-a301-9073d502a86a nodeName:}" failed. No retries permitted until 2026-10-30 21:00:21.773667953 +0000 UTC m=+8.894741680 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q9bnp" (UID: "b4ef3fe7-a7fb-473c-a301-9073d502a86a") : secret "insights-runtime-extractor-tls" not found Oct 30 21:00:17.773887 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:17.773632 4926 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 21:00:17.773887 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:17.773714 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert podName:9ffe8e19-6888-4eca-ae9d-9d0fc05ba752 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:21.773703297 +0000 UTC m=+8.894777039 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-wbrhm" (UID: "9ffe8e19-6888-4eca-ae9d-9d0fc05ba752") : secret "networking-console-plugin-cert" not found Oct 30 21:00:17.773887 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:17.773730 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs podName:f6337834-db43-4b8e-a955-9691f72e0f19 nodeName:}" failed. No retries permitted until 2026-10-30 21:00:21.773721087 +0000 UTC m=+8.894794819 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs") pod "network-metrics-daemon-7d6gs" (UID: "f6337834-db43-4b8e-a955-9691f72e0f19") : secret "metrics-daemon-secret" not found Oct 30 21:00:17.773887 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:17.773745 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle podName:276d4076-cd5a-45d6-b4f8-2de14f9c9a0d nodeName:}" failed. No retries permitted until 2026-10-30 21:00:21.773736509 +0000 UTC m=+8.894810232 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle") pod "router-default-6789dbc4b5-5gqgl" (UID: "276d4076-cd5a-45d6-b4f8-2de14f9c9a0d") : configmap references non-existent config key: service-ca.crt Oct 30 21:00:17.773887 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:17.773766 4926 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 21:00:17.773887 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:17.773776 4926 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-56886949f8-xwhft: secret "image-registry-tls" not found Oct 30 21:00:17.773887 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:17.773809 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls podName:9b61b643-c3cf-453c-875e-25576445ad4f nodeName:}" failed. No retries permitted until 2026-10-30 21:00:21.773798198 +0000 UTC m=+8.894871934 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls") pod "image-registry-56886949f8-xwhft" (UID: "9b61b643-c3cf-453c-875e-25576445ad4f") : secret "image-registry-tls" not found Oct 30 21:00:17.776053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:17.776016 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/662ead00-b524-4a6e-815b-c3399b3c9e6c-metrics-tls\") pod \"dns-default-888xx\" (UID: \"662ead00-b524-4a6e-815b-c3399b3c9e6c\") " pod="openshift-dns/dns-default-888xx" Oct 30 21:00:17.821946 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:17.821912 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:17.822085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:17.822044 4926 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:00:17.827094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:17.827072 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-c5khs" Oct 30 21:00:17.883318 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:17.883296 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:17.931794 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:17.931768 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-888xx" Oct 30 21:00:18.231915 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:18.231886 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-888xx"] Oct 30 21:00:18.265476 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:18.265446 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662ead00_b524_4a6e_815b_c3399b3c9e6c.slice/crio-320888aead4203e0b6e1018c857a663e668070985cb2188e511561936f527eb5 WatchSource:0}: Error finding container 320888aead4203e0b6e1018c857a663e668070985cb2188e511561936f527eb5: Status 404 returned error can't find the container with id 320888aead4203e0b6e1018c857a663e668070985cb2188e511561936f527eb5 Oct 30 21:00:19.092487 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:19.092445 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-888xx" event={"ID":"662ead00-b524-4a6e-815b-c3399b3c9e6c","Type":"ContainerStarted","Data":"320888aead4203e0b6e1018c857a663e668070985cb2188e511561936f527eb5"} Oct 30 21:00:19.094130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:19.094104 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-v5rrz" event={"ID":"0b77d4c7-37eb-4bad-8367-31346289726e","Type":"ContainerStarted","Data":"c42c74678511bae15408c0f772227ff552bce7e2a5bbdf6c7a78b7474a821fbe"} Oct 30 21:00:19.095691 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:19.095665 4926 generic.go:356] "Generic (PLEG): container finished" podID="ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6" containerID="14e178e155cb5c8f968b53dd6215ca77ba5047023842b774dab62b80f5246ba1" exitCode=0 Oct 30 21:00:19.095781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:19.095726 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-m7m4c" event={"ID":"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6","Type":"ContainerDied","Data":"14e178e155cb5c8f968b53dd6215ca77ba5047023842b774dab62b80f5246ba1"} Oct 30 21:00:19.096086 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:19.096069 4926 scope.go:122] "RemoveContainer" containerID="14e178e155cb5c8f968b53dd6215ca77ba5047023842b774dab62b80f5246ba1" Oct 30 21:00:20.100451 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:20.100365 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-888xx" event={"ID":"662ead00-b524-4a6e-815b-c3399b3c9e6c","Type":"ContainerStarted","Data":"41159113d6a88367bbb17c688cb5615da8f12a24a40ce9a46410d5e4b187a1d3"} Oct 30 21:00:20.100451 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:20.100422 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-888xx" event={"ID":"662ead00-b524-4a6e-815b-c3399b3c9e6c","Type":"ContainerStarted","Data":"202fd7de82280fcdb540920de2a46e3cfc72c820bb17a2a2e71f5647cb907320"} Oct 30 21:00:20.100938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:20.100473 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-888xx" Oct 30 21:00:20.102324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:20.102296 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-m7m4c" event={"ID":"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6","Type":"ContainerStarted","Data":"9c374dc7f704e40aa00b58290656f078bcebba172f2e37e0a3e085d7530720b7"} Oct 30 21:00:20.445579 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:20.445551 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:20.445812 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:20.445800 4926 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:00:21.812893 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:21.812858 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:21.813307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:21.812899 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wbrhm\" (UID: \"9ffe8e19-6888-4eca-ae9d-9d0fc05ba752\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" Oct 30 21:00:21.813307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:21.812920 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:21.813307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:21.812951 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs\") pod \"network-metrics-daemon-7d6gs\" (UID: \"f6337834-db43-4b8e-a955-9691f72e0f19\") " pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 21:00:21.813307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:21.812996 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:21.813307 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:21.813001 4926 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:00:21.813307 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:21.813092 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls podName:b4ef3fe7-a7fb-473c-a301-9073d502a86a nodeName:}" failed. No retries permitted until 2026-10-30 21:00:29.813072902 +0000 UTC m=+16.934146644 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-q9bnp" (UID: "b4ef3fe7-a7fb-473c-a301-9073d502a86a") : secret "insights-runtime-extractor-tls" not found Oct 30 21:00:21.813563 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:21.813369 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:21.813689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:21.813668 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/276d4076-cd5a-45d6-b4f8-2de14f9c9a0d-service-ca-bundle\") pod \"router-default-6789dbc4b5-5gqgl\" (UID: \"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d\") " pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:21.815301 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:21.815281 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6337834-db43-4b8e-a955-9691f72e0f19-metrics-certs\") pod \"network-metrics-daemon-7d6gs\" (UID: \"f6337834-db43-4b8e-a955-9691f72e0f19\") " pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 21:00:21.815621 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:21.815597 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls\") pod \"image-registry-758fcd4795-j6bts\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:21.815693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:21.815633 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls\") pod \"image-registry-56886949f8-xwhft\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:21.815746 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:21.815723 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9ffe8e19-6888-4eca-ae9d-9d0fc05ba752-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-wbrhm\" (UID: \"9ffe8e19-6888-4eca-ae9d-9d0fc05ba752\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" Oct 30 21:00:21.820572 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:21.820551 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:21.822722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:21.822541 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:21.822722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:21.822621 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" Oct 30 21:00:21.830388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:21.830367 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7d6gs" Oct 30 21:00:21.831452 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:21.831434 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:22.146525 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:22.146444 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-wbrhm"] Oct 30 21:00:22.150869 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:22.150845 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffe8e19_6888_4eca_ae9d_9d0fc05ba752.slice/crio-6860f5699aca03339e3e2a9481b088927709ffa0d643ecfd1b1d28d8349e3c03 WatchSource:0}: Error finding container 6860f5699aca03339e3e2a9481b088927709ffa0d643ecfd1b1d28d8349e3c03: Status 404 returned error can't find the container with id 6860f5699aca03339e3e2a9481b088927709ffa0d643ecfd1b1d28d8349e3c03 Oct 30 21:00:22.157310 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:22.157287 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7d6gs"] Oct 30 21:00:22.160230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:22.160207 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56886949f8-xwhft"] Oct 30 21:00:22.160995 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:22.160974 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6337834_db43_4b8e_a955_9691f72e0f19.slice/crio-786c136b460347753592a1d176b19a1fafdc0ab3c000b3a9d787d9ba07733fc2 WatchSource:0}: Error finding container 786c136b460347753592a1d176b19a1fafdc0ab3c000b3a9d787d9ba07733fc2: Status 404 returned error can't find the container with id 786c136b460347753592a1d176b19a1fafdc0ab3c000b3a9d787d9ba07733fc2 Oct 30 21:00:22.164154 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:22.164105 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b61b643_c3cf_453c_875e_25576445ad4f.slice/crio-f49e749e1d93bf46e545d5ce3e81e6aed8cfb1c49978823c4b5ee74215d2f0ef WatchSource:0}: Error finding container f49e749e1d93bf46e545d5ce3e81e6aed8cfb1c49978823c4b5ee74215d2f0ef: Status 404 returned error can't find the container with id f49e749e1d93bf46e545d5ce3e81e6aed8cfb1c49978823c4b5ee74215d2f0ef Oct 30 21:00:22.171013 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:22.170991 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6789dbc4b5-5gqgl"] Oct 30 21:00:22.173947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:22.173925 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-758fcd4795-j6bts"] Oct 30 21:00:22.174803 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:22.174778 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod276d4076_cd5a_45d6_b4f8_2de14f9c9a0d.slice/crio-42ce538a3d6133f59f3dd850d130e5acebeb57fb9562d840710d8d736c127542 WatchSource:0}: Error finding container 42ce538a3d6133f59f3dd850d130e5acebeb57fb9562d840710d8d736c127542: Status 404 returned error can't find the container with id 42ce538a3d6133f59f3dd850d130e5acebeb57fb9562d840710d8d736c127542 Oct 30 21:00:22.178172 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:22.178100 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b4ba52c_ad04_4a25_9158_89941367abd3.slice/crio-fed368225861839a1e8710a2e9555531d84d58f3f3ff6ee4702cb37a67417e66 WatchSource:0}: Error finding container fed368225861839a1e8710a2e9555531d84d58f3f3ff6ee4702cb37a67417e66: Status 404 returned error can't find the container with id fed368225861839a1e8710a2e9555531d84d58f3f3ff6ee4702cb37a67417e66 Oct 30 21:00:23.115149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:23.115112 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7d6gs" event={"ID":"f6337834-db43-4b8e-a955-9691f72e0f19","Type":"ContainerStarted","Data":"786c136b460347753592a1d176b19a1fafdc0ab3c000b3a9d787d9ba07733fc2"} Oct 30 21:00:23.116441 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:23.116414 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" event={"ID":"9ffe8e19-6888-4eca-ae9d-9d0fc05ba752","Type":"ContainerStarted","Data":"6860f5699aca03339e3e2a9481b088927709ffa0d643ecfd1b1d28d8349e3c03"} Oct 30 21:00:23.117802 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:23.117773 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" event={"ID":"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d","Type":"ContainerStarted","Data":"01a91958a0474be69cb696d04d86f072d4d79adb56fabc6159b72f40806d1d4d"} Oct 30 21:00:23.117887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:23.117819 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" event={"ID":"276d4076-cd5a-45d6-b4f8-2de14f9c9a0d","Type":"ContainerStarted","Data":"42ce538a3d6133f59f3dd850d130e5acebeb57fb9562d840710d8d736c127542"} Oct 30 21:00:23.119100 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:23.119078 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-758fcd4795-j6bts" event={"ID":"6b4ba52c-ad04-4a25-9158-89941367abd3","Type":"ContainerStarted","Data":"b97d590c3747b1979cf3bab5d44d966a6fc5a67330242cb0e641cc28f1be05e7"} Oct 30 21:00:23.119186 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:23.119116 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-758fcd4795-j6bts" event={"ID":"6b4ba52c-ad04-4a25-9158-89941367abd3","Type":"ContainerStarted","Data":"fed368225861839a1e8710a2e9555531d84d58f3f3ff6ee4702cb37a67417e66"} Oct 30 21:00:23.120434 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:23.120414 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56886949f8-xwhft" event={"ID":"9b61b643-c3cf-453c-875e-25576445ad4f","Type":"ContainerStarted","Data":"06d786cf2be1a85f9334dc0da6dd03788c5ba5fcfb5b44883a9517ab3703a960"} Oct 30 21:00:23.120527 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:23.120450 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56886949f8-xwhft" event={"ID":"9b61b643-c3cf-453c-875e-25576445ad4f","Type":"ContainerStarted","Data":"f49e749e1d93bf46e545d5ce3e81e6aed8cfb1c49978823c4b5ee74215d2f0ef"} Oct 30 21:00:23.168474 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:23.168440 4926 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ffe8e19_6888_4eca_ae9d_9d0fc05ba752.slice/crio-f292f00c832f2581baa96a6db00924213425d1992d2f31f3fc20f34eadd9daca.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:00:23.524202 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:23.524170 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mgsr5" Oct 30 21:00:23.823781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:23.823708 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:23.826309 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:23.826287 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:24.126926 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:24.126836 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7d6gs" event={"ID":"f6337834-db43-4b8e-a955-9691f72e0f19","Type":"ContainerStarted","Data":"1e9e433d13eb8940ee315690000ed7d6367b41bf3efb2eb6c5af1c0d7dcb300a"} Oct 30 21:00:24.127376 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:24.126925 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7d6gs" event={"ID":"f6337834-db43-4b8e-a955-9691f72e0f19","Type":"ContainerStarted","Data":"cf6e6a3deb5ca43f0bfaac1dc284b969f3cdba443c039b7c7d3b0809988b391d"} Oct 30 21:00:24.128517 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:24.128497 4926 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:00:24.128517 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:24.128495 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-wbrhm" event={"ID":"9ffe8e19-6888-4eca-ae9d-9d0fc05ba752","Type":"ContainerStarted","Data":"f292f00c832f2581baa96a6db00924213425d1992d2f31f3fc20f34eadd9daca"} Oct 30 21:00:24.128793 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:24.128773 4926 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:00:25.132722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:25.132690 4926 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:00:26.883232 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:26.883198 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:28.994043 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:28.993997 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:28.994525 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:28.994180 4926 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:00:28.995328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:28.995308 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6789dbc4b5-5gqgl" Oct 30 21:00:29.886523 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:29.886470 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:29.888829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:29.888801 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b4ef3fe7-a7fb-473c-a301-9073d502a86a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-q9bnp\" (UID: \"b4ef3fe7-a7fb-473c-a301-9073d502a86a\") " pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:29.933828 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:29.933794 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-q9bnp" Oct 30 21:00:30.108445 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:30.108411 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-888xx" Oct 30 21:00:30.227530 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:30.227067 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-q9bnp"] Oct 30 21:00:30.231046 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:30.231001 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ef3fe7_a7fb_473c_a301_9073d502a86a.slice/crio-527cc0d872b073fa61b2f6df6e28b84c71e19d51c9d4c2ce193ff5620e2cdb26 WatchSource:0}: Error finding container 527cc0d872b073fa61b2f6df6e28b84c71e19d51c9d4c2ce193ff5620e2cdb26: Status 404 returned error can't find the container with id 527cc0d872b073fa61b2f6df6e28b84c71e19d51c9d4c2ce193ff5620e2cdb26 Oct 30 21:00:30.243703 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:30.243680 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:30.622641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:30.622500 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56886949f8-xwhft"] Oct 30 21:00:31.154593 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:31.154498 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q9bnp" event={"ID":"b4ef3fe7-a7fb-473c-a301-9073d502a86a","Type":"ContainerStarted","Data":"190d5202c10b6539722755c21e198c9b41ddec0b4370094c7e8c62b00ab77def"} Oct 30 21:00:31.154593 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:31.154566 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q9bnp" event={"ID":"b4ef3fe7-a7fb-473c-a301-9073d502a86a","Type":"ContainerStarted","Data":"486846da1c728aac0475c4938fa3c8b73d48075eb949ad83591f1f96cb94588c"} Oct 30 21:00:31.155023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:31.154597 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q9bnp" event={"ID":"b4ef3fe7-a7fb-473c-a301-9073d502a86a","Type":"ContainerStarted","Data":"527cc0d872b073fa61b2f6df6e28b84c71e19d51c9d4c2ce193ff5620e2cdb26"} Oct 30 21:00:31.759921 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:31.759872 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vtgj9" Oct 30 21:00:32.163690 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:32.163654 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-q9bnp" event={"ID":"b4ef3fe7-a7fb-473c-a301-9073d502a86a","Type":"ContainerStarted","Data":"466f2c8cd0d8a817e240e8059854afcd079eb09402d00e0c260b5540040df685"} Oct 30 21:00:32.369644 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:32.369569 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-888xx_662ead00-b524-4a6e-815b-c3399b3c9e6c/dns/0.log" Oct 30 21:00:32.571648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:32.571616 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-888xx_662ead00-b524-4a6e-815b-c3399b3c9e6c/kube-rbac-proxy/0.log" Oct 30 21:00:33.169582 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:33.169556 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-54qxj_a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31/dns-node-resolver/0.log" Oct 30 21:00:33.769840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:33.769809 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-56886949f8-xwhft_9b61b643-c3cf-453c-875e-25576445ad4f/registry/0.log" Oct 30 21:00:33.969074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:33.969047 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-758fcd4795-j6bts_6b4ba52c-ad04-4a25-9158-89941367abd3/registry/0.log" Oct 30 21:00:34.369257 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:34.369228 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cnj4q_1362c712-7332-4a6e-a215-633d2f02acca/node-ca/0.log" Oct 30 21:00:34.969635 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:34.969605 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6789dbc4b5-5gqgl_276d4076-cd5a-45d6-b4f8-2de14f9c9a0d/router/0.log" Oct 30 21:00:35.569351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:35.569325 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mmzzs_4901add7-3312-4950-ae4c-4d6581380e9e/serve-healthcheck-canary/0.log" Oct 30 21:00:37.482881 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:37.482842 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:00:40.077171 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.077129 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s24tx"] Oct 30 21:00:40.106568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.106538 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.137776 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.137743 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:00:40.138150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.138129 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:00:40.138246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.138208 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-595gg\"" Oct 30 21:00:40.162778 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.162744 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.162945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.162789 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dd2h\" (UniqueName: \"kubernetes.io/projected/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-kube-api-access-7dd2h\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.162945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.162876 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-node-exporter-tls\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.162945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.162912 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-node-exporter-textfile\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.162945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.162939 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-root\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.163130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.162953 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-metrics-client-ca\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.163130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.163056 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-sys\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.163130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.163097 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-node-exporter-wtmp\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.163130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.163124 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-node-exporter-accelerators-collector-config\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.173234 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.173206 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:00:40.173344 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.173283 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:00:40.264484 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.264449 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.264484 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.264489 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dd2h\" (UniqueName: \"kubernetes.io/projected/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-kube-api-access-7dd2h\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.264767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.264520 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-node-exporter-tls\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.264767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.264542 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-node-exporter-textfile\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.264767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.264561 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-root\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.264767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.264578 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-metrics-client-ca\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.264767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.264740 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-root\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.264767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.264749 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-sys\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.265084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.264794 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-node-exporter-wtmp\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.265084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.264835 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-node-exporter-accelerators-collector-config\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.265084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.264795 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-sys\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.265084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.264908 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-node-exporter-wtmp\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.265084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.264987 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-node-exporter-textfile\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.265367 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.265346 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-metrics-client-ca\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.265477 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.265444 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-node-exporter-accelerators-collector-config\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.266944 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.266916 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-node-exporter-tls\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.267107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.267090 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.282461 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.282435 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dd2h\" (UniqueName: \"kubernetes.io/projected/34ad4f2a-37ae-4b6c-968c-34a4369fe36c-kube-api-access-7dd2h\") pod \"node-exporter-s24tx\" (UID: \"34ad4f2a-37ae-4b6c-968c-34a4369fe36c\") " pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.459444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:40.459413 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s24tx" Oct 30 21:00:40.498996 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:00:40.498952 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ad4f2a_37ae_4b6c_968c_34a4369fe36c.slice/crio-6387905a6d85bef7320ceebd48278de2f7b5523233b1ce161a2f3a88b730e7bc WatchSource:0}: Error finding container 6387905a6d85bef7320ceebd48278de2f7b5523233b1ce161a2f3a88b730e7bc: Status 404 returned error can't find the container with id 6387905a6d85bef7320ceebd48278de2f7b5523233b1ce161a2f3a88b730e7bc Oct 30 21:00:41.162465 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:41.162431 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:41.200615 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:41.200580 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s24tx" event={"ID":"34ad4f2a-37ae-4b6c-968c-34a4369fe36c","Type":"ContainerStarted","Data":"6387905a6d85bef7320ceebd48278de2f7b5523233b1ce161a2f3a88b730e7bc"} Oct 30 21:00:42.206192 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:42.206156 4926 generic.go:356] "Generic (PLEG): container finished" podID="34ad4f2a-37ae-4b6c-968c-34a4369fe36c" containerID="814e85624e048048ed72a28116cfda089924df06198cf6a2c969df07fb73f139" exitCode=0 Oct 30 21:00:42.206580 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:42.206244 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s24tx" event={"ID":"34ad4f2a-37ae-4b6c-968c-34a4369fe36c","Type":"ContainerDied","Data":"814e85624e048048ed72a28116cfda089924df06198cf6a2c969df07fb73f139"} Oct 30 21:00:43.213565 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:43.213522 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s24tx" event={"ID":"34ad4f2a-37ae-4b6c-968c-34a4369fe36c","Type":"ContainerStarted","Data":"5032abff47798f9739e7926477b44333643a84e122e7f5e32075b771cfba4c43"} Oct 30 21:00:43.214165 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:43.213592 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s24tx" event={"ID":"34ad4f2a-37ae-4b6c-968c-34a4369fe36c","Type":"ContainerStarted","Data":"8c787e861be2c8732350aa95d5a11725a0ccd5b3d12eac100db396316a28b30e"} Oct 30 21:00:43.274221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:43.274170 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s24tx" podStartSLOduration=1.8502149129999999 podStartE2EDuration="3.274155886s" totalImagesPullingTime="1.345279226s" totalInitContainerRuntime="78.661747ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:00:43.262862539 +0000 UTC m=+30.383936285" watchObservedRunningTime="2026-10-30 21:00:43.274155886 +0000 UTC m=+30.395229631" Oct 30 21:00:46.881999 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:46.881967 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mfgjb" Oct 30 21:00:46.888259 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:46.888235 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:00:53.559950 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:53.559915 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-758fcd4795-j6bts"] Oct 30 21:00:56.173781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.173538 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-56886949f8-xwhft" podUID="9b61b643-c3cf-453c-875e-25576445ad4f" containerName="registry" containerID="cri-o://06d786cf2be1a85f9334dc0da6dd03788c5ba5fcfb5b44883a9517ab3703a960" gracePeriod=30 Oct 30 21:00:56.433863 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.433812 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:56.510670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.510639 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b61b643-c3cf-453c-875e-25576445ad4f-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b61b643-c3cf-453c-875e-25576445ad4f-installation-pull-secrets\") pod \"9b61b643-c3cf-453c-875e-25576445ad4f\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " Oct 30 21:00:56.510847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.510681 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b61b643-c3cf-453c-875e-25576445ad4f-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b61b643-c3cf-453c-875e-25576445ad4f-registry-certificates\") pod \"9b61b643-c3cf-453c-875e-25576445ad4f\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " Oct 30 21:00:56.510847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.510715 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b61b643-c3cf-453c-875e-25576445ad4f-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b61b643-c3cf-453c-875e-25576445ad4f-image-registry-private-configuration\") pod \"9b61b643-c3cf-453c-875e-25576445ad4f\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " Oct 30 21:00:56.510847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.510747 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b61b643-c3cf-453c-875e-25576445ad4f-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b61b643-c3cf-453c-875e-25576445ad4f-ca-trust-extracted\") pod \"9b61b643-c3cf-453c-875e-25576445ad4f\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " Oct 30 21:00:56.510847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.510767 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-bound-sa-token\") pod \"9b61b643-c3cf-453c-875e-25576445ad4f\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " Oct 30 21:00:56.510847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.510786 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-kube-api-access-rhvqf\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-kube-api-access-rhvqf\") pod \"9b61b643-c3cf-453c-875e-25576445ad4f\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " Oct 30 21:00:56.510847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.510820 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b61b643-c3cf-453c-875e-25576445ad4f-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b61b643-c3cf-453c-875e-25576445ad4f-trusted-ca\") pod \"9b61b643-c3cf-453c-875e-25576445ad4f\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " Oct 30 21:00:56.511228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.510885 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls\") pod \"9b61b643-c3cf-453c-875e-25576445ad4f\" (UID: \"9b61b643-c3cf-453c-875e-25576445ad4f\") " Oct 30 21:00:56.511228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.511198 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b61b643-c3cf-453c-875e-25576445ad4f-registry-certificates" pod "9b61b643-c3cf-453c-875e-25576445ad4f" (UID: "9b61b643-c3cf-453c-875e-25576445ad4f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:56.511564 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.511517 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b61b643-c3cf-453c-875e-25576445ad4f-trusted-ca" pod "9b61b643-c3cf-453c-875e-25576445ad4f" (UID: "9b61b643-c3cf-453c-875e-25576445ad4f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:00:56.514199 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.514165 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-bound-sa-token" pod "9b61b643-c3cf-453c-875e-25576445ad4f" (UID: "9b61b643-c3cf-453c-875e-25576445ad4f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:56.514490 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.514470 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls" pod "9b61b643-c3cf-453c-875e-25576445ad4f" (UID: "9b61b643-c3cf-453c-875e-25576445ad4f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:56.514566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.514465 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b61b643-c3cf-453c-875e-25576445ad4f-image-registry-private-configuration" pod "9b61b643-c3cf-453c-875e-25576445ad4f" (UID: "9b61b643-c3cf-453c-875e-25576445ad4f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:56.514615 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.514590 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b61b643-c3cf-453c-875e-25576445ad4f-installation-pull-secrets" pod "9b61b643-c3cf-453c-875e-25576445ad4f" (UID: "9b61b643-c3cf-453c-875e-25576445ad4f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:00:56.514801 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.514775 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-kube-api-access-rhvqf" pod "9b61b643-c3cf-453c-875e-25576445ad4f" (UID: "9b61b643-c3cf-453c-875e-25576445ad4f"). InnerVolumeSpecName "kube-api-access-rhvqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:00:56.518283 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.518258 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b61b643-c3cf-453c-875e-25576445ad4f-ca-trust-extracted" pod "9b61b643-c3cf-453c-875e-25576445ad4f" (UID: "9b61b643-c3cf-453c-875e-25576445ad4f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:00:56.612193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.612149 4926 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9b61b643-c3cf-453c-875e-25576445ad4f-installation-pull-secrets\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:00:56.612193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.612189 4926 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9b61b643-c3cf-453c-875e-25576445ad4f-registry-certificates\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:00:56.612193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.612200 4926 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9b61b643-c3cf-453c-875e-25576445ad4f-image-registry-private-configuration\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:00:56.612423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.612212 4926 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9b61b643-c3cf-453c-875e-25576445ad4f-ca-trust-extracted\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:00:56.612423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.612222 4926 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-bound-sa-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:00:56.612423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.612231 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhvqf\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-kube-api-access-rhvqf\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:00:56.612423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.612239 4926 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9b61b643-c3cf-453c-875e-25576445ad4f-trusted-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:00:56.612423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:56.612248 4926 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9b61b643-c3cf-453c-875e-25576445ad4f-registry-tls\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:00:57.277361 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:57.277273 4926 generic.go:356] "Generic (PLEG): container finished" podID="9b61b643-c3cf-453c-875e-25576445ad4f" containerID="06d786cf2be1a85f9334dc0da6dd03788c5ba5fcfb5b44883a9517ab3703a960" exitCode=0 Oct 30 21:00:57.277361 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:57.277327 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56886949f8-xwhft" Oct 30 21:00:57.277889 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:57.277358 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56886949f8-xwhft" event={"ID":"9b61b643-c3cf-453c-875e-25576445ad4f","Type":"ContainerDied","Data":"06d786cf2be1a85f9334dc0da6dd03788c5ba5fcfb5b44883a9517ab3703a960"} Oct 30 21:00:57.277889 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:57.277411 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56886949f8-xwhft" event={"ID":"9b61b643-c3cf-453c-875e-25576445ad4f","Type":"ContainerDied","Data":"f49e749e1d93bf46e545d5ce3e81e6aed8cfb1c49978823c4b5ee74215d2f0ef"} Oct 30 21:00:57.277889 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:57.277435 4926 scope.go:122] "RemoveContainer" containerID="06d786cf2be1a85f9334dc0da6dd03788c5ba5fcfb5b44883a9517ab3703a960" Oct 30 21:00:57.285665 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:57.285648 4926 scope.go:122] "RemoveContainer" containerID="06d786cf2be1a85f9334dc0da6dd03788c5ba5fcfb5b44883a9517ab3703a960" Oct 30 21:00:57.285951 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:00:57.285931 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06d786cf2be1a85f9334dc0da6dd03788c5ba5fcfb5b44883a9517ab3703a960\": container with ID starting with 06d786cf2be1a85f9334dc0da6dd03788c5ba5fcfb5b44883a9517ab3703a960 not found: ID does not exist" containerID="06d786cf2be1a85f9334dc0da6dd03788c5ba5fcfb5b44883a9517ab3703a960" Oct 30 21:00:57.285999 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:57.285962 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d786cf2be1a85f9334dc0da6dd03788c5ba5fcfb5b44883a9517ab3703a960"} err="failed to get container status \"06d786cf2be1a85f9334dc0da6dd03788c5ba5fcfb5b44883a9517ab3703a960\": rpc error: code = NotFound desc = could not find container \"06d786cf2be1a85f9334dc0da6dd03788c5ba5fcfb5b44883a9517ab3703a960\": container with ID starting with 06d786cf2be1a85f9334dc0da6dd03788c5ba5fcfb5b44883a9517ab3703a960 not found: ID does not exist" Oct 30 21:00:57.335206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:57.335173 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-56886949f8-xwhft"] Oct 30 21:00:57.346589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:57.346564 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-56886949f8-xwhft"] Oct 30 21:00:59.013311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:00:59.013277 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b61b643-c3cf-453c-875e-25576445ad4f" path="/var/lib/kubelet/pods/9b61b643-c3cf-453c-875e-25576445ad4f/volumes" Oct 30 21:01:07.482200 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:01:07.482166 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:01:18.579879 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:18.579838 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-758fcd4795-j6bts" podUID="6b4ba52c-ad04-4a25-9158-89941367abd3" containerName="registry" containerID="cri-o://b97d590c3747b1979cf3bab5d44d966a6fc5a67330242cb0e641cc28f1be05e7" gracePeriod=30 Oct 30 21:01:18.821763 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:18.821740 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:01:18.986817 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:18.986775 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-bound-sa-token\") pod \"6b4ba52c-ad04-4a25-9158-89941367abd3\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " Oct 30 21:01:18.986988 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:18.986857 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b4ba52c-ad04-4a25-9158-89941367abd3-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b4ba52c-ad04-4a25-9158-89941367abd3-installation-pull-secrets\") pod \"6b4ba52c-ad04-4a25-9158-89941367abd3\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " Oct 30 21:01:18.986988 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:18.986945 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-certificates\") pod \"6b4ba52c-ad04-4a25-9158-89941367abd3\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " Oct 30 21:01:18.987099 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:18.986985 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-kube-api-access-vrt6b\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-kube-api-access-vrt6b\") pod \"6b4ba52c-ad04-4a25-9158-89941367abd3\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " Oct 30 21:01:18.987175 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:18.987147 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b4ba52c-ad04-4a25-9158-89941367abd3-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b4ba52c-ad04-4a25-9158-89941367abd3-ca-trust-extracted\") pod \"6b4ba52c-ad04-4a25-9158-89941367abd3\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " Oct 30 21:01:18.987265 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:18.987251 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls\") pod \"6b4ba52c-ad04-4a25-9158-89941367abd3\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " Oct 30 21:01:18.987342 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:18.987325 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b4ba52c-ad04-4a25-9158-89941367abd3-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b4ba52c-ad04-4a25-9158-89941367abd3-image-registry-private-configuration\") pod \"6b4ba52c-ad04-4a25-9158-89941367abd3\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " Oct 30 21:01:18.987396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:18.987371 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b4ba52c-ad04-4a25-9158-89941367abd3-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b4ba52c-ad04-4a25-9158-89941367abd3-trusted-ca\") pod \"6b4ba52c-ad04-4a25-9158-89941367abd3\" (UID: \"6b4ba52c-ad04-4a25-9158-89941367abd3\") " Oct 30 21:01:18.987506 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:18.987476 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-certificates" pod "6b4ba52c-ad04-4a25-9158-89941367abd3" (UID: "6b4ba52c-ad04-4a25-9158-89941367abd3"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:01:18.987610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:18.987589 4926 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-certificates\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:01:18.987917 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:18.987853 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b4ba52c-ad04-4a25-9158-89941367abd3-trusted-ca" pod "6b4ba52c-ad04-4a25-9158-89941367abd3" (UID: "6b4ba52c-ad04-4a25-9158-89941367abd3"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:01:18.989513 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:18.989457 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b4ba52c-ad04-4a25-9158-89941367abd3-image-registry-private-configuration" pod "6b4ba52c-ad04-4a25-9158-89941367abd3" (UID: "6b4ba52c-ad04-4a25-9158-89941367abd3"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:01:18.989618 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:18.989561 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-bound-sa-token" pod "6b4ba52c-ad04-4a25-9158-89941367abd3" (UID: "6b4ba52c-ad04-4a25-9158-89941367abd3"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:18.989705 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:18.989684 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b4ba52c-ad04-4a25-9158-89941367abd3-installation-pull-secrets" pod "6b4ba52c-ad04-4a25-9158-89941367abd3" (UID: "6b4ba52c-ad04-4a25-9158-89941367abd3"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:01:18.989780 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:18.989763 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-kube-api-access-vrt6b" pod "6b4ba52c-ad04-4a25-9158-89941367abd3" (UID: "6b4ba52c-ad04-4a25-9158-89941367abd3"). InnerVolumeSpecName "kube-api-access-vrt6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:18.989848 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:18.989763 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls" pod "6b4ba52c-ad04-4a25-9158-89941367abd3" (UID: "6b4ba52c-ad04-4a25-9158-89941367abd3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:01:18.996290 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:18.996209 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b4ba52c-ad04-4a25-9158-89941367abd3-ca-trust-extracted" pod "6b4ba52c-ad04-4a25-9158-89941367abd3" (UID: "6b4ba52c-ad04-4a25-9158-89941367abd3"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:01:19.088837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:19.088808 4926 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6b4ba52c-ad04-4a25-9158-89941367abd3-installation-pull-secrets\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:01:19.088837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:19.088834 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrt6b\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-kube-api-access-vrt6b\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:01:19.088976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:19.088848 4926 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6b4ba52c-ad04-4a25-9158-89941367abd3-ca-trust-extracted\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:01:19.088976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:19.088861 4926 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-registry-tls\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:01:19.088976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:19.088875 4926 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6b4ba52c-ad04-4a25-9158-89941367abd3-image-registry-private-configuration\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:01:19.088976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:19.088889 4926 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6b4ba52c-ad04-4a25-9158-89941367abd3-trusted-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:01:19.088976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:19.088901 4926 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6b4ba52c-ad04-4a25-9158-89941367abd3-bound-sa-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:01:19.363589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:19.363507 4926 generic.go:356] "Generic (PLEG): container finished" podID="6b4ba52c-ad04-4a25-9158-89941367abd3" containerID="b97d590c3747b1979cf3bab5d44d966a6fc5a67330242cb0e641cc28f1be05e7" exitCode=0 Oct 30 21:01:19.363589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:19.363562 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-758fcd4795-j6bts" Oct 30 21:01:19.363808 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:19.363593 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-758fcd4795-j6bts" event={"ID":"6b4ba52c-ad04-4a25-9158-89941367abd3","Type":"ContainerDied","Data":"b97d590c3747b1979cf3bab5d44d966a6fc5a67330242cb0e641cc28f1be05e7"} Oct 30 21:01:19.363808 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:19.363644 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-758fcd4795-j6bts" event={"ID":"6b4ba52c-ad04-4a25-9158-89941367abd3","Type":"ContainerDied","Data":"fed368225861839a1e8710a2e9555531d84d58f3f3ff6ee4702cb37a67417e66"} Oct 30 21:01:19.363808 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:19.363664 4926 scope.go:122] "RemoveContainer" containerID="b97d590c3747b1979cf3bab5d44d966a6fc5a67330242cb0e641cc28f1be05e7" Oct 30 21:01:19.371655 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:19.371625 4926 scope.go:122] "RemoveContainer" containerID="b97d590c3747b1979cf3bab5d44d966a6fc5a67330242cb0e641cc28f1be05e7" Oct 30 21:01:19.371921 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:01:19.371899 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b97d590c3747b1979cf3bab5d44d966a6fc5a67330242cb0e641cc28f1be05e7\": container with ID starting with b97d590c3747b1979cf3bab5d44d966a6fc5a67330242cb0e641cc28f1be05e7 not found: ID does not exist" containerID="b97d590c3747b1979cf3bab5d44d966a6fc5a67330242cb0e641cc28f1be05e7" Oct 30 21:01:19.371975 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:19.371932 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b97d590c3747b1979cf3bab5d44d966a6fc5a67330242cb0e641cc28f1be05e7"} err="failed to get container status \"b97d590c3747b1979cf3bab5d44d966a6fc5a67330242cb0e641cc28f1be05e7\": rpc error: code = NotFound desc = could not find container \"b97d590c3747b1979cf3bab5d44d966a6fc5a67330242cb0e641cc28f1be05e7\": container with ID starting with b97d590c3747b1979cf3bab5d44d966a6fc5a67330242cb0e641cc28f1be05e7 not found: ID does not exist" Oct 30 21:01:19.420594 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:19.420569 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-758fcd4795-j6bts"] Oct 30 21:01:19.431485 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:19.431462 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-758fcd4795-j6bts"] Oct 30 21:01:21.013188 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:21.013155 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b4ba52c-ad04-4a25-9158-89941367abd3" path="/var/lib/kubelet/pods/6b4ba52c-ad04-4a25-9158-89941367abd3/volumes" Oct 30 21:01:23.380918 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:23.380884 4926 generic.go:356] "Generic (PLEG): container finished" podID="4ce43826-7550-4c62-8acc-48714541e426" containerID="0de05589dcb53d97bf066d86d8415cbe1973c30429448da0a7c744f7c6553995" exitCode=0 Oct 30 21:01:23.381290 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:23.380954 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7" event={"ID":"4ce43826-7550-4c62-8acc-48714541e426","Type":"ContainerDied","Data":"0de05589dcb53d97bf066d86d8415cbe1973c30429448da0a7c744f7c6553995"} Oct 30 21:01:23.381290 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:23.381284 4926 scope.go:122] "RemoveContainer" containerID="0de05589dcb53d97bf066d86d8415cbe1973c30429448da0a7c744f7c6553995" Oct 30 21:01:24.386849 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:24.386813 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-frsk7" event={"ID":"4ce43826-7550-4c62-8acc-48714541e426","Type":"ContainerStarted","Data":"e1b3c30474feb220a702b547bd2cd3e5b290806a4262be997d7fd721987097af"} Oct 30 21:01:35.428783 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:35.428748 4926 generic.go:356] "Generic (PLEG): container finished" podID="ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6" containerID="9c374dc7f704e40aa00b58290656f078bcebba172f2e37e0a3e085d7530720b7" exitCode=0 Oct 30 21:01:35.429344 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:35.428827 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-m7m4c" event={"ID":"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6","Type":"ContainerDied","Data":"9c374dc7f704e40aa00b58290656f078bcebba172f2e37e0a3e085d7530720b7"} Oct 30 21:01:35.429344 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:35.428884 4926 scope.go:122] "RemoveContainer" containerID="14e178e155cb5c8f968b53dd6215ca77ba5047023842b774dab62b80f5246ba1" Oct 30 21:01:35.429344 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:35.429248 4926 scope.go:122] "RemoveContainer" containerID="9c374dc7f704e40aa00b58290656f078bcebba172f2e37e0a3e085d7530720b7" Oct 30 21:01:35.429503 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:01:35.429470 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-m7m4c_openshift-insights(ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6)\"" pod="openshift-insights/insights-operator-7c4986d449-m7m4c" podUID="ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6" Oct 30 21:01:37.482087 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:01:37.482048 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:01:43.462026 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:43.461989 4926 generic.go:356] "Generic (PLEG): container finished" podID="705e2b0e-10e1-4710-8dfb-bbd862c8c084" containerID="a3f9d7ad7953e493dcad1e9a8cb28b83bf31a3796b2aa6f324b2572a42ece688" exitCode=0 Oct 30 21:01:43.462498 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:43.462062 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm" event={"ID":"705e2b0e-10e1-4710-8dfb-bbd862c8c084","Type":"ContainerDied","Data":"a3f9d7ad7953e493dcad1e9a8cb28b83bf31a3796b2aa6f324b2572a42ece688"} Oct 30 21:01:43.462498 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:43.462416 4926 scope.go:122] "RemoveContainer" containerID="a3f9d7ad7953e493dcad1e9a8cb28b83bf31a3796b2aa6f324b2572a42ece688" Oct 30 21:01:44.467353 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:44.467318 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vprbm" event={"ID":"705e2b0e-10e1-4710-8dfb-bbd862c8c084","Type":"ContainerStarted","Data":"b26d219eba568fd4dbfe5054a7ad765b0e1efbf087ca6aa8e7c636c9b104c4a8"} Oct 30 21:01:47.008545 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:47.008506 4926 scope.go:122] "RemoveContainer" containerID="9c374dc7f704e40aa00b58290656f078bcebba172f2e37e0a3e085d7530720b7" Oct 30 21:01:47.480746 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:01:47.480709 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-m7m4c" event={"ID":"ae5a8c83-7a0c-4f91-a504-eda2e3e6bdb6","Type":"ContainerStarted","Data":"acb90ee7b33ef97519ed46b435601fa4d3aecf8159b658499f1e715e90764b78"} Oct 30 21:02:07.481260 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:02:07.481224 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:02:37.481817 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:02:37.481740 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:03:07.481581 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:03:07.481544 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:03:37.482498 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:03:37.482451 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:03:56.524368 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.524295 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9"] Oct 30 21:03:56.524750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.524648 4926 cpu_manager.go:414] "containerMap: removing container" podUID="9b61b643-c3cf-453c-875e-25576445ad4f" containerName="registry" Oct 30 21:03:56.524750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.524662 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b61b643-c3cf-453c-875e-25576445ad4f" containerName="registry" Oct 30 21:03:56.524750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.524677 4926 cpu_manager.go:414] "containerMap: removing container" podUID="6b4ba52c-ad04-4a25-9158-89941367abd3" containerName="registry" Oct 30 21:03:56.524750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.524682 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b4ba52c-ad04-4a25-9158-89941367abd3" containerName="registry" Oct 30 21:03:56.524750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.524726 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b61b643-c3cf-453c-875e-25576445ad4f" containerName="registry" Oct 30 21:03:56.524750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.524736 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b4ba52c-ad04-4a25-9158-89941367abd3" containerName="registry" Oct 30 21:03:56.527793 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.527777 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9" Oct 30 21:03:56.543267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.543245 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 21:03:56.554076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.554055 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 21:03:56.554281 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.554261 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-vplgz\"" Oct 30 21:03:56.555754 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.555733 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9"] Oct 30 21:03:56.640798 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.640760 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7d2fe4e-4aff-498f-b177-75668fcac436-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9\" (UID: \"c7d2fe4e-4aff-498f-b177-75668fcac436\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9" Oct 30 21:03:56.640798 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.640799 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94xr4\" (UniqueName: \"kubernetes.io/projected/c7d2fe4e-4aff-498f-b177-75668fcac436-kube-api-access-94xr4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9\" (UID: \"c7d2fe4e-4aff-498f-b177-75668fcac436\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9" Oct 30 21:03:56.640998 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.640934 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7d2fe4e-4aff-498f-b177-75668fcac436-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9\" (UID: \"c7d2fe4e-4aff-498f-b177-75668fcac436\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9" Oct 30 21:03:56.703629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.703590 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk"] Oct 30 21:03:56.707042 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.707013 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk" Oct 30 21:03:56.720740 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.720718 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-k9v7j\"" Oct 30 21:03:56.726062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.726023 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk"] Oct 30 21:03:56.742452 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.742301 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7d2fe4e-4aff-498f-b177-75668fcac436-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9\" (UID: \"c7d2fe4e-4aff-498f-b177-75668fcac436\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9" Oct 30 21:03:56.742452 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.742337 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94xr4\" (UniqueName: \"kubernetes.io/projected/c7d2fe4e-4aff-498f-b177-75668fcac436-kube-api-access-94xr4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9\" (UID: \"c7d2fe4e-4aff-498f-b177-75668fcac436\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9" Oct 30 21:03:56.742592 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.742547 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7d2fe4e-4aff-498f-b177-75668fcac436-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9\" (UID: \"c7d2fe4e-4aff-498f-b177-75668fcac436\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9" Oct 30 21:03:56.742663 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.742640 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7d2fe4e-4aff-498f-b177-75668fcac436-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9\" (UID: \"c7d2fe4e-4aff-498f-b177-75668fcac436\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9" Oct 30 21:03:56.742857 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.742837 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7d2fe4e-4aff-498f-b177-75668fcac436-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9\" (UID: \"c7d2fe4e-4aff-498f-b177-75668fcac436\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9" Oct 30 21:03:56.761395 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.761366 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94xr4\" (UniqueName: \"kubernetes.io/projected/c7d2fe4e-4aff-498f-b177-75668fcac436-kube-api-access-94xr4\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9\" (UID: \"c7d2fe4e-4aff-498f-b177-75668fcac436\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9" Oct 30 21:03:56.842923 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.842849 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9" Oct 30 21:03:56.843083 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.843015 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/567c5bbf-74c2-4d88-b460-53c5ca846f70-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk\" (UID: \"567c5bbf-74c2-4d88-b460-53c5ca846f70\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk" Oct 30 21:03:56.843149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.843114 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwkqg\" (UniqueName: \"kubernetes.io/projected/567c5bbf-74c2-4d88-b460-53c5ca846f70-kube-api-access-rwkqg\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk\" (UID: \"567c5bbf-74c2-4d88-b460-53c5ca846f70\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk" Oct 30 21:03:56.843149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.843142 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/567c5bbf-74c2-4d88-b460-53c5ca846f70-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk\" (UID: \"567c5bbf-74c2-4d88-b460-53c5ca846f70\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk" Oct 30 21:03:56.943643 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.943608 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/567c5bbf-74c2-4d88-b460-53c5ca846f70-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk\" (UID: \"567c5bbf-74c2-4d88-b460-53c5ca846f70\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk" Oct 30 21:03:56.943643 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.943646 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwkqg\" (UniqueName: \"kubernetes.io/projected/567c5bbf-74c2-4d88-b460-53c5ca846f70-kube-api-access-rwkqg\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk\" (UID: \"567c5bbf-74c2-4d88-b460-53c5ca846f70\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk" Oct 30 21:03:56.943884 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.943665 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/567c5bbf-74c2-4d88-b460-53c5ca846f70-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk\" (UID: \"567c5bbf-74c2-4d88-b460-53c5ca846f70\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk" Oct 30 21:03:56.943991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.943971 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/567c5bbf-74c2-4d88-b460-53c5ca846f70-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk\" (UID: \"567c5bbf-74c2-4d88-b460-53c5ca846f70\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk" Oct 30 21:03:56.944067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.943999 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/567c5bbf-74c2-4d88-b460-53c5ca846f70-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk\" (UID: \"567c5bbf-74c2-4d88-b460-53c5ca846f70\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk" Oct 30 21:03:56.965603 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:56.965571 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwkqg\" (UniqueName: \"kubernetes.io/projected/567c5bbf-74c2-4d88-b460-53c5ca846f70-kube-api-access-rwkqg\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk\" (UID: \"567c5bbf-74c2-4d88-b460-53c5ca846f70\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk" Oct 30 21:03:57.022222 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:57.022197 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk" Oct 30 21:03:57.137458 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:57.137386 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9"] Oct 30 21:03:57.141094 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:03:57.141064 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d2fe4e_4aff_498f_b177_75668fcac436.slice/crio-7ba1940ce68c23209797ef0728e3ba10f717b764b9fb3f22e04c3cb65425a0a7 WatchSource:0}: Error finding container 7ba1940ce68c23209797ef0728e3ba10f717b764b9fb3f22e04c3cb65425a0a7: Status 404 returned error can't find the container with id 7ba1940ce68c23209797ef0728e3ba10f717b764b9fb3f22e04c3cb65425a0a7 Oct 30 21:03:57.311232 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:57.311202 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk"] Oct 30 21:03:57.314421 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:03:57.314389 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567c5bbf_74c2_4d88_b460_53c5ca846f70.slice/crio-118f4e42dacabcbbd959345070f09b6b3f3f491ef3128a87a119c71c5c208958 WatchSource:0}: Error finding container 118f4e42dacabcbbd959345070f09b6b3f3f491ef3128a87a119c71c5c208958: Status 404 returned error can't find the container with id 118f4e42dacabcbbd959345070f09b6b3f3f491ef3128a87a119c71c5c208958 Oct 30 21:03:57.942776 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:57.942736 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk" event={"ID":"567c5bbf-74c2-4d88-b460-53c5ca846f70","Type":"ContainerStarted","Data":"118f4e42dacabcbbd959345070f09b6b3f3f491ef3128a87a119c71c5c208958"} Oct 30 21:03:57.944719 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:57.944662 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9" event={"ID":"c7d2fe4e-4aff-498f-b177-75668fcac436","Type":"ContainerStarted","Data":"7ba1940ce68c23209797ef0728e3ba10f717b764b9fb3f22e04c3cb65425a0a7"} Oct 30 21:03:59.309620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:59.309581 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q"] Oct 30 21:03:59.313543 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:59.313521 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q" Oct 30 21:03:59.341864 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:59.341843 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-dc8s5\"" Oct 30 21:03:59.343001 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:59.342976 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q"] Oct 30 21:03:59.467116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:59.467077 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06816f46-272f-4772-9c69-f3af428dbba7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q\" (UID: \"06816f46-272f-4772-9c69-f3af428dbba7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q" Oct 30 21:03:59.467280 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:59.467127 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swbzh\" (UniqueName: \"kubernetes.io/projected/06816f46-272f-4772-9c69-f3af428dbba7-kube-api-access-swbzh\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q\" (UID: \"06816f46-272f-4772-9c69-f3af428dbba7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q" Oct 30 21:03:59.467280 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:59.467223 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06816f46-272f-4772-9c69-f3af428dbba7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q\" (UID: \"06816f46-272f-4772-9c69-f3af428dbba7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q" Oct 30 21:03:59.568677 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:59.568588 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06816f46-272f-4772-9c69-f3af428dbba7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q\" (UID: \"06816f46-272f-4772-9c69-f3af428dbba7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q" Oct 30 21:03:59.568677 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:59.568640 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swbzh\" (UniqueName: \"kubernetes.io/projected/06816f46-272f-4772-9c69-f3af428dbba7-kube-api-access-swbzh\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q\" (UID: \"06816f46-272f-4772-9c69-f3af428dbba7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q" Oct 30 21:03:59.568914 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:59.568679 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06816f46-272f-4772-9c69-f3af428dbba7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q\" (UID: \"06816f46-272f-4772-9c69-f3af428dbba7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q" Oct 30 21:03:59.569416 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:59.569068 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06816f46-272f-4772-9c69-f3af428dbba7-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q\" (UID: \"06816f46-272f-4772-9c69-f3af428dbba7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q" Oct 30 21:03:59.569416 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:59.569120 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06816f46-272f-4772-9c69-f3af428dbba7-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q\" (UID: \"06816f46-272f-4772-9c69-f3af428dbba7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q" Oct 30 21:03:59.588567 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:59.588524 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swbzh\" (UniqueName: \"kubernetes.io/projected/06816f46-272f-4772-9c69-f3af428dbba7-kube-api-access-swbzh\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q\" (UID: \"06816f46-272f-4772-9c69-f3af428dbba7\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q" Oct 30 21:03:59.634568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:59.634534 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q" Oct 30 21:03:59.935114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:59.935083 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q"] Oct 30 21:03:59.939198 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:03:59.939158 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06816f46_272f_4772_9c69_f3af428dbba7.slice/crio-54df055841ea85eb9735f9e7333f2d9914b712ac875586b227c184c575ec4ee7 WatchSource:0}: Error finding container 54df055841ea85eb9735f9e7333f2d9914b712ac875586b227c184c575ec4ee7: Status 404 returned error can't find the container with id 54df055841ea85eb9735f9e7333f2d9914b712ac875586b227c184c575ec4ee7 Oct 30 21:03:59.954489 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:03:59.954459 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q" event={"ID":"06816f46-272f-4772-9c69-f3af428dbba7","Type":"ContainerStarted","Data":"54df055841ea85eb9735f9e7333f2d9914b712ac875586b227c184c575ec4ee7"} Oct 30 21:04:00.807176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:00.807143 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8"] Oct 30 21:04:00.810989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:00.810962 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8" Oct 30 21:04:00.838680 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:00.838651 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-srztk\"" Oct 30 21:04:00.839922 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:00.839897 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8"] Oct 30 21:04:00.882049 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:00.882000 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e2c013c-ba60-4b4d-a37b-7656c6e228b5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8\" (UID: \"0e2c013c-ba60-4b4d-a37b-7656c6e228b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8" Oct 30 21:04:00.882236 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:00.882119 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvf6k\" (UniqueName: \"kubernetes.io/projected/0e2c013c-ba60-4b4d-a37b-7656c6e228b5-kube-api-access-hvf6k\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8\" (UID: \"0e2c013c-ba60-4b4d-a37b-7656c6e228b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8" Oct 30 21:04:00.882236 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:00.882168 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e2c013c-ba60-4b4d-a37b-7656c6e228b5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8\" (UID: \"0e2c013c-ba60-4b4d-a37b-7656c6e228b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8" Oct 30 21:04:00.898980 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:00.898953 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj"] Oct 30 21:04:00.903930 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:00.903565 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj" Oct 30 21:04:00.922359 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:00.922332 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-bvvgv\"" Oct 30 21:04:00.925922 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:00.925015 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj"] Oct 30 21:04:00.983693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:00.983655 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e2c013c-ba60-4b4d-a37b-7656c6e228b5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8\" (UID: \"0e2c013c-ba60-4b4d-a37b-7656c6e228b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8" Oct 30 21:04:00.983860 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:00.983708 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvf6k\" (UniqueName: \"kubernetes.io/projected/0e2c013c-ba60-4b4d-a37b-7656c6e228b5-kube-api-access-hvf6k\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8\" (UID: \"0e2c013c-ba60-4b4d-a37b-7656c6e228b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8" Oct 30 21:04:00.983860 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:00.983742 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e2c013c-ba60-4b4d-a37b-7656c6e228b5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8\" (UID: \"0e2c013c-ba60-4b4d-a37b-7656c6e228b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8" Oct 30 21:04:00.983860 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:00.983773 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nrwl\" (UniqueName: \"kubernetes.io/projected/8ebb7012-3a31-42e0-9bc1-21a72e2a8140-kube-api-access-5nrwl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj\" (UID: \"8ebb7012-3a31-42e0-9bc1-21a72e2a8140\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj" Oct 30 21:04:00.983860 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:00.983817 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ebb7012-3a31-42e0-9bc1-21a72e2a8140-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj\" (UID: \"8ebb7012-3a31-42e0-9bc1-21a72e2a8140\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj" Oct 30 21:04:00.984095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:00.983880 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ebb7012-3a31-42e0-9bc1-21a72e2a8140-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj\" (UID: \"8ebb7012-3a31-42e0-9bc1-21a72e2a8140\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj" Oct 30 21:04:00.984369 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:00.984341 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e2c013c-ba60-4b4d-a37b-7656c6e228b5-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8\" (UID: \"0e2c013c-ba60-4b4d-a37b-7656c6e228b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8" Oct 30 21:04:00.984955 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:00.984910 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e2c013c-ba60-4b4d-a37b-7656c6e228b5-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8\" (UID: \"0e2c013c-ba60-4b4d-a37b-7656c6e228b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8" Oct 30 21:04:01.017843 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.017788 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvf6k\" (UniqueName: \"kubernetes.io/projected/0e2c013c-ba60-4b4d-a37b-7656c6e228b5-kube-api-access-hvf6k\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8\" (UID: \"0e2c013c-ba60-4b4d-a37b-7656c6e228b5\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8" Oct 30 21:04:01.021294 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.021268 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc"] Oct 30 21:04:01.025754 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.025734 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc" Oct 30 21:04:01.049628 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.049581 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-f9x25\"" Oct 30 21:04:01.054135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.054072 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc"] Oct 30 21:04:01.085554 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.084753 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ebb7012-3a31-42e0-9bc1-21a72e2a8140-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj\" (UID: \"8ebb7012-3a31-42e0-9bc1-21a72e2a8140\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj" Oct 30 21:04:01.085554 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.084809 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nrwl\" (UniqueName: \"kubernetes.io/projected/8ebb7012-3a31-42e0-9bc1-21a72e2a8140-kube-api-access-5nrwl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj\" (UID: \"8ebb7012-3a31-42e0-9bc1-21a72e2a8140\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj" Oct 30 21:04:01.085554 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.084836 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ebb7012-3a31-42e0-9bc1-21a72e2a8140-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj\" (UID: \"8ebb7012-3a31-42e0-9bc1-21a72e2a8140\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj" Oct 30 21:04:01.085554 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.085222 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ebb7012-3a31-42e0-9bc1-21a72e2a8140-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj\" (UID: \"8ebb7012-3a31-42e0-9bc1-21a72e2a8140\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj" Oct 30 21:04:01.085554 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.085417 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ebb7012-3a31-42e0-9bc1-21a72e2a8140-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj\" (UID: \"8ebb7012-3a31-42e0-9bc1-21a72e2a8140\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj" Oct 30 21:04:01.130701 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.130641 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz"] Oct 30 21:04:01.134536 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.131890 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8" Oct 30 21:04:01.134536 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.134487 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nrwl\" (UniqueName: \"kubernetes.io/projected/8ebb7012-3a31-42e0-9bc1-21a72e2a8140-kube-api-access-5nrwl\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj\" (UID: \"8ebb7012-3a31-42e0-9bc1-21a72e2a8140\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj" Oct 30 21:04:01.137963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.137945 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz" Oct 30 21:04:01.154109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.154073 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-g5xq9\"" Oct 30 21:04:01.160576 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.160498 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz"] Oct 30 21:04:01.186054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.185799 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljnxm\" (UniqueName: \"kubernetes.io/projected/52e53d48-9908-477d-98f5-af9739c6b346-kube-api-access-ljnxm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc\" (UID: \"52e53d48-9908-477d-98f5-af9739c6b346\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc" Oct 30 21:04:01.186054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.185882 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52e53d48-9908-477d-98f5-af9739c6b346-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc\" (UID: \"52e53d48-9908-477d-98f5-af9739c6b346\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc" Oct 30 21:04:01.186054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.185933 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52e53d48-9908-477d-98f5-af9739c6b346-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc\" (UID: \"52e53d48-9908-477d-98f5-af9739c6b346\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc" Oct 30 21:04:01.236303 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.236276 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj" Oct 30 21:04:01.287557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.286384 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee4ef697-79cd-452e-be1c-20a8a92d51c8-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz\" (UID: \"ee4ef697-79cd-452e-be1c-20a8a92d51c8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz" Oct 30 21:04:01.287557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.286449 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx2rw\" (UniqueName: \"kubernetes.io/projected/ee4ef697-79cd-452e-be1c-20a8a92d51c8-kube-api-access-wx2rw\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz\" (UID: \"ee4ef697-79cd-452e-be1c-20a8a92d51c8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz" Oct 30 21:04:01.287557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.286487 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljnxm\" (UniqueName: \"kubernetes.io/projected/52e53d48-9908-477d-98f5-af9739c6b346-kube-api-access-ljnxm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc\" (UID: \"52e53d48-9908-477d-98f5-af9739c6b346\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc" Oct 30 21:04:01.287557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.286538 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52e53d48-9908-477d-98f5-af9739c6b346-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc\" (UID: \"52e53d48-9908-477d-98f5-af9739c6b346\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc" Oct 30 21:04:01.287557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.286560 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee4ef697-79cd-452e-be1c-20a8a92d51c8-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz\" (UID: \"ee4ef697-79cd-452e-be1c-20a8a92d51c8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz" Oct 30 21:04:01.287557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.286590 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52e53d48-9908-477d-98f5-af9739c6b346-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc\" (UID: \"52e53d48-9908-477d-98f5-af9739c6b346\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc" Oct 30 21:04:01.287557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.286943 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52e53d48-9908-477d-98f5-af9739c6b346-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc\" (UID: \"52e53d48-9908-477d-98f5-af9739c6b346\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc" Oct 30 21:04:01.287557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.287503 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52e53d48-9908-477d-98f5-af9739c6b346-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc\" (UID: \"52e53d48-9908-477d-98f5-af9739c6b346\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc" Oct 30 21:04:01.309168 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.309139 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljnxm\" (UniqueName: \"kubernetes.io/projected/52e53d48-9908-477d-98f5-af9739c6b346-kube-api-access-ljnxm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc\" (UID: \"52e53d48-9908-477d-98f5-af9739c6b346\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc" Oct 30 21:04:01.345000 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.344933 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc" Oct 30 21:04:01.388600 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.388061 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee4ef697-79cd-452e-be1c-20a8a92d51c8-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz\" (UID: \"ee4ef697-79cd-452e-be1c-20a8a92d51c8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz" Oct 30 21:04:01.388600 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.388141 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee4ef697-79cd-452e-be1c-20a8a92d51c8-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz\" (UID: \"ee4ef697-79cd-452e-be1c-20a8a92d51c8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz" Oct 30 21:04:01.388600 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.388190 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx2rw\" (UniqueName: \"kubernetes.io/projected/ee4ef697-79cd-452e-be1c-20a8a92d51c8-kube-api-access-wx2rw\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz\" (UID: \"ee4ef697-79cd-452e-be1c-20a8a92d51c8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz" Oct 30 21:04:01.389340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.389116 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee4ef697-79cd-452e-be1c-20a8a92d51c8-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz\" (UID: \"ee4ef697-79cd-452e-be1c-20a8a92d51c8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz" Oct 30 21:04:01.389437 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.388925 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee4ef697-79cd-452e-be1c-20a8a92d51c8-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz\" (UID: \"ee4ef697-79cd-452e-be1c-20a8a92d51c8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz" Oct 30 21:04:01.417500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.417469 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx2rw\" (UniqueName: \"kubernetes.io/projected/ee4ef697-79cd-452e-be1c-20a8a92d51c8-kube-api-access-wx2rw\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz\" (UID: \"ee4ef697-79cd-452e-be1c-20a8a92d51c8\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz" Oct 30 21:04:01.458565 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.458525 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8"] Oct 30 21:04:01.463635 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:04:01.463599 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2c013c_ba60_4b4d_a37b_7656c6e228b5.slice/crio-93be3de74511fa7efb0388cd28707311345229f6c309aa0882fed6541acf809e WatchSource:0}: Error finding container 93be3de74511fa7efb0388cd28707311345229f6c309aa0882fed6541acf809e: Status 404 returned error can't find the container with id 93be3de74511fa7efb0388cd28707311345229f6c309aa0882fed6541acf809e Oct 30 21:04:01.476772 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.476745 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz" Oct 30 21:04:01.485109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.485079 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq"] Oct 30 21:04:01.489152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.489127 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.511938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.511911 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq"] Oct 30 21:04:01.512795 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.512773 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:04:01.527253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.527228 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-w59m6\"" Oct 30 21:04:01.559622 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.559233 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj"] Oct 30 21:04:01.573407 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:04:01.573375 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ebb7012_3a31_42e0_9bc1_21a72e2a8140.slice/crio-e088edc66aee95c94807d6c2a83877c68be1e6fef0b973adf06a458072759882 WatchSource:0}: Error finding container e088edc66aee95c94807d6c2a83877c68be1e6fef0b973adf06a458072759882: Status 404 returned error can't find the container with id e088edc66aee95c94807d6c2a83877c68be1e6fef0b973adf06a458072759882 Oct 30 21:04:01.590314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.590279 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.590454 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.590337 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.590454 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.590384 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.590454 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.590419 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.590669 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.590458 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.590669 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.590507 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sftbv\" (UniqueName: \"kubernetes.io/projected/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-kube-api-access-sftbv\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.590669 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.590537 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.590669 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.590566 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.590669 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.590607 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.619612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.617898 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b"] Oct 30 21:04:01.621965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.621940 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.639644 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.639615 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-n6tgl\"" Oct 30 21:04:01.643642 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.643582 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b"] Oct 30 21:04:01.646941 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.646913 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc"] Oct 30 21:04:01.652247 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:04:01.652222 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e53d48_9908_477d_98f5_af9739c6b346.slice/crio-395b3d833d4cef4fa862360284518ca713ca096c6bc2c9fac2669400a0330496 WatchSource:0}: Error finding container 395b3d833d4cef4fa862360284518ca713ca096c6bc2c9fac2669400a0330496: Status 404 returned error can't find the container with id 395b3d833d4cef4fa862360284518ca713ca096c6bc2c9fac2669400a0330496 Oct 30 21:04:01.691338 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.691297 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3d0842d-2236-4cad-ad5a-f1c47655c5be-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.691486 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.691352 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.691486 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.691390 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.691486 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.691414 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d0842d-2236-4cad-ad5a-f1c47655c5be-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.691486 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.691442 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3d0842d-2236-4cad-ad5a-f1c47655c5be-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.691486 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.691480 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.691744 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.691524 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.691744 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.691561 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3d0842d-2236-4cad-ad5a-f1c47655c5be-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.691744 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.691594 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.691744 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.691628 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.691744 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.691655 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d0842d-2236-4cad-ad5a-f1c47655c5be-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.691744 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.691683 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3d0842d-2236-4cad-ad5a-f1c47655c5be-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.691744 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.691723 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.692092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.691758 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3d0842d-2236-4cad-ad5a-f1c47655c5be-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.692092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.691792 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcd8w\" (UniqueName: \"kubernetes.io/projected/b3d0842d-2236-4cad-ad5a-f1c47655c5be-kube-api-access-mcd8w\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.692092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.691831 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3d0842d-2236-4cad-ad5a-f1c47655c5be-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.692092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.691864 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sftbv\" (UniqueName: \"kubernetes.io/projected/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-kube-api-access-sftbv\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.692092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.691894 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.692397 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.692336 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.693329 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.692619 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.693329 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.693253 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.693507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.693446 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.693921 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.693895 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.696670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.696643 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.697137 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.697099 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.711719 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.711663 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.712430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.712378 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sftbv\" (UniqueName: \"kubernetes.io/projected/d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94-kube-api-access-sftbv\") pod \"maas-default-gateway-openshift-default-57f78bcd55-wxstq\" (UID: \"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.792132 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.792006 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz"] Oct 30 21:04:01.792428 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.792399 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3d0842d-2236-4cad-ad5a-f1c47655c5be-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.792546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.792456 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d0842d-2236-4cad-ad5a-f1c47655c5be-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.792546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.792486 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3d0842d-2236-4cad-ad5a-f1c47655c5be-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.792546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.792527 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3d0842d-2236-4cad-ad5a-f1c47655c5be-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.792713 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.792564 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcd8w\" (UniqueName: \"kubernetes.io/projected/b3d0842d-2236-4cad-ad5a-f1c47655c5be-kube-api-access-mcd8w\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.792713 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.792599 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3d0842d-2236-4cad-ad5a-f1c47655c5be-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.792713 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.792634 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3d0842d-2236-4cad-ad5a-f1c47655c5be-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.792713 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.792666 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d0842d-2236-4cad-ad5a-f1c47655c5be-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.792713 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.792691 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3d0842d-2236-4cad-ad5a-f1c47655c5be-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.793484 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.793464 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3d0842d-2236-4cad-ad5a-f1c47655c5be-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.793773 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.793690 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3d0842d-2236-4cad-ad5a-f1c47655c5be-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.794503 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.793938 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d0842d-2236-4cad-ad5a-f1c47655c5be-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.794503 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.794211 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3d0842d-2236-4cad-ad5a-f1c47655c5be-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.794641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.794529 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3d0842d-2236-4cad-ad5a-f1c47655c5be-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.796200 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:04:01.796169 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee4ef697_79cd_452e_be1c_20a8a92d51c8.slice/crio-c356061ab35788cd68fe0e97cd26095ceeeb7b243d750be62c7ecae2b5ad5931 WatchSource:0}: Error finding container c356061ab35788cd68fe0e97cd26095ceeeb7b243d750be62c7ecae2b5ad5931: Status 404 returned error can't find the container with id c356061ab35788cd68fe0e97cd26095ceeeb7b243d750be62c7ecae2b5ad5931 Oct 30 21:04:01.797585 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.797544 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3d0842d-2236-4cad-ad5a-f1c47655c5be-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.798293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.798251 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3d0842d-2236-4cad-ad5a-f1c47655c5be-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.811293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.811268 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3d0842d-2236-4cad-ad5a-f1c47655c5be-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.812069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.812012 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcd8w\" (UniqueName: \"kubernetes.io/projected/b3d0842d-2236-4cad-ad5a-f1c47655c5be-kube-api-access-mcd8w\") pod \"openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b\" (UID: \"b3d0842d-2236-4cad-ad5a-f1c47655c5be\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.815374 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.815162 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:01.939167 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.939096 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:01.970163 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.970124 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc" event={"ID":"52e53d48-9908-477d-98f5-af9739c6b346","Type":"ContainerStarted","Data":"395b3d833d4cef4fa862360284518ca713ca096c6bc2c9fac2669400a0330496"} Oct 30 21:04:01.974753 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.974713 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8" event={"ID":"0e2c013c-ba60-4b4d-a37b-7656c6e228b5","Type":"ContainerStarted","Data":"93be3de74511fa7efb0388cd28707311345229f6c309aa0882fed6541acf809e"} Oct 30 21:04:01.978336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.978224 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj" event={"ID":"8ebb7012-3a31-42e0-9bc1-21a72e2a8140","Type":"ContainerStarted","Data":"e088edc66aee95c94807d6c2a83877c68be1e6fef0b973adf06a458072759882"} Oct 30 21:04:01.979719 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:01.979645 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz" event={"ID":"ee4ef697-79cd-452e-be1c-20a8a92d51c8","Type":"ContainerStarted","Data":"c356061ab35788cd68fe0e97cd26095ceeeb7b243d750be62c7ecae2b5ad5931"} Oct 30 21:04:02.150069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:02.149916 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq"] Oct 30 21:04:02.153797 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:04:02.153759 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9bc6398_ec05_4cb3_8ba6_3dcfc5b9cb94.slice/crio-eb9fb68db2e36a81f776b63821c4b062c7f93f0f9e8cbed484e08f8abc79e5b3 WatchSource:0}: Error finding container eb9fb68db2e36a81f776b63821c4b062c7f93f0f9e8cbed484e08f8abc79e5b3: Status 404 returned error can't find the container with id eb9fb68db2e36a81f776b63821c4b062c7f93f0f9e8cbed484e08f8abc79e5b3 Oct 30 21:04:02.261749 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:02.261717 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b"] Oct 30 21:04:02.272454 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:04:02.272419 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d0842d_2236_4cad_ad5a_f1c47655c5be.slice/crio-8519637a0746b9e7f6d76adf1401447c1be0e546e3769cf69966332248de1534 WatchSource:0}: Error finding container 8519637a0746b9e7f6d76adf1401447c1be0e546e3769cf69966332248de1534: Status 404 returned error can't find the container with id 8519637a0746b9e7f6d76adf1401447c1be0e546e3769cf69966332248de1534 Oct 30 21:04:02.994291 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:02.994249 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" event={"ID":"b3d0842d-2236-4cad-ad5a-f1c47655c5be","Type":"ContainerStarted","Data":"8519637a0746b9e7f6d76adf1401447c1be0e546e3769cf69966332248de1534"} Oct 30 21:04:02.998799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:02.998774 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" event={"ID":"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94","Type":"ContainerStarted","Data":"eb9fb68db2e36a81f776b63821c4b062c7f93f0f9e8cbed484e08f8abc79e5b3"} Oct 30 21:04:03.001964 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:03.001934 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q"] Oct 30 21:04:03.006164 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:03.006142 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q" Oct 30 21:04:03.033973 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:03.033939 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q"] Oct 30 21:04:03.119057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:03.118268 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n975b\" (UniqueName: \"kubernetes.io/projected/17568512-7448-46a4-ad28-fc64c94d36fa-kube-api-access-n975b\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q\" (UID: \"17568512-7448-46a4-ad28-fc64c94d36fa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q" Oct 30 21:04:03.119057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:03.118385 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17568512-7448-46a4-ad28-fc64c94d36fa-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q\" (UID: \"17568512-7448-46a4-ad28-fc64c94d36fa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q" Oct 30 21:04:03.119057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:03.118413 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17568512-7448-46a4-ad28-fc64c94d36fa-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q\" (UID: \"17568512-7448-46a4-ad28-fc64c94d36fa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q" Oct 30 21:04:03.219060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:03.219008 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n975b\" (UniqueName: \"kubernetes.io/projected/17568512-7448-46a4-ad28-fc64c94d36fa-kube-api-access-n975b\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q\" (UID: \"17568512-7448-46a4-ad28-fc64c94d36fa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q" Oct 30 21:04:03.219241 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:03.219129 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17568512-7448-46a4-ad28-fc64c94d36fa-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q\" (UID: \"17568512-7448-46a4-ad28-fc64c94d36fa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q" Oct 30 21:04:03.219241 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:03.219157 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17568512-7448-46a4-ad28-fc64c94d36fa-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q\" (UID: \"17568512-7448-46a4-ad28-fc64c94d36fa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q" Oct 30 21:04:03.219622 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:03.219596 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17568512-7448-46a4-ad28-fc64c94d36fa-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q\" (UID: \"17568512-7448-46a4-ad28-fc64c94d36fa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q" Oct 30 21:04:03.219716 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:03.219662 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17568512-7448-46a4-ad28-fc64c94d36fa-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q\" (UID: \"17568512-7448-46a4-ad28-fc64c94d36fa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q" Oct 30 21:04:03.251972 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:03.251889 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n975b\" (UniqueName: \"kubernetes.io/projected/17568512-7448-46a4-ad28-fc64c94d36fa-kube-api-access-n975b\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q\" (UID: \"17568512-7448-46a4-ad28-fc64c94d36fa\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q" Oct 30 21:04:03.337320 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:03.337276 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q" Oct 30 21:04:03.697150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:03.696883 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q"] Oct 30 21:04:04.017943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:04.017812 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q" event={"ID":"17568512-7448-46a4-ad28-fc64c94d36fa","Type":"ContainerStarted","Data":"6051ea6b8c2064f02d94884e4cbbb6a38fd4e47a1659a6902a7c10c7e9b73035"} Oct 30 21:04:07.505814 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:04:07.505713 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:04:09.080952 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:09.079839 4926 generic.go:356] "Generic (PLEG): container finished" podID="c7d2fe4e-4aff-498f-b177-75668fcac436" containerID="0aac5350a29d5a29c4ed6ca7ed3865b874c7e56d843d061a755a8a5c1bd86b0c" exitCode=0 Oct 30 21:04:09.080952 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:09.080010 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9" event={"ID":"c7d2fe4e-4aff-498f-b177-75668fcac436","Type":"ContainerDied","Data":"0aac5350a29d5a29c4ed6ca7ed3865b874c7e56d843d061a755a8a5c1bd86b0c"} Oct 30 21:04:09.096975 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:09.095155 4926 generic.go:356] "Generic (PLEG): container finished" podID="567c5bbf-74c2-4d88-b460-53c5ca846f70" containerID="4b13d6f640f3e73921c1f51ef6a00fc718f2b31b94776f4b6c0a6d8784851cb0" exitCode=0 Oct 30 21:04:09.096975 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:09.095202 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk" event={"ID":"567c5bbf-74c2-4d88-b460-53c5ca846f70","Type":"ContainerDied","Data":"4b13d6f640f3e73921c1f51ef6a00fc718f2b31b94776f4b6c0a6d8784851cb0"} Oct 30 21:04:10.112932 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:10.112847 4926 generic.go:356] "Generic (PLEG): container finished" podID="567c5bbf-74c2-4d88-b460-53c5ca846f70" containerID="5b689ca4d77cbde6e80657057de7fda78425e4d40227b7581ad384bafe0c0f54" exitCode=0 Oct 30 21:04:10.113414 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:10.112945 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk" event={"ID":"567c5bbf-74c2-4d88-b460-53c5ca846f70","Type":"ContainerDied","Data":"5b689ca4d77cbde6e80657057de7fda78425e4d40227b7581ad384bafe0c0f54"} Oct 30 21:04:10.599297 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:10.599259 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:04:10.599413 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:10.599351 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:04:10.599413 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:10.599396 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:04:10.907008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:10.906970 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:04:10.907142 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:10.907077 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:04:10.907142 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:10.907128 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:04:11.127492 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:11.126926 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" event={"ID":"b3d0842d-2236-4cad-ad5a-f1c47655c5be","Type":"ContainerStarted","Data":"a142159498b65fd6de59e171f6128f625142262bb01bbc5bd56e000367054e38"} Oct 30 21:04:11.136172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:11.136086 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" event={"ID":"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94","Type":"ContainerStarted","Data":"7278eeb91f6d7e18c8681619c0bfdcbd1223cc8f3b6d6f78667ff277680dedca"} Oct 30 21:04:11.182066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:11.181374 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" podStartSLOduration=1.551581488 podStartE2EDuration="10.181359142s" totalImagesPullingTime="8.629777654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:11.179620435 +0000 UTC m=+238.300694181" watchObservedRunningTime="2026-10-30 21:04:11.181359142 +0000 UTC m=+238.302432887" Oct 30 21:04:11.816102 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:11.816052 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:11.822152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:11.821893 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:11.863838 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:11.862944 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" podStartSLOduration=2.421751751 podStartE2EDuration="10.862926813s" totalImagesPullingTime="8.441175062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:11.219412932 +0000 UTC m=+238.340486680" watchObservedRunningTime="2026-10-30 21:04:11.862926813 +0000 UTC m=+238.984000562" Oct 30 21:04:11.940405 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:11.940294 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:12.154322 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:12.153229 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj" event={"ID":"8ebb7012-3a31-42e0-9bc1-21a72e2a8140","Type":"ContainerStarted","Data":"b088883c17276465f7bbc20c440177f44d533e61ac37d5e3eafdf100f1b97bcf"} Oct 30 21:04:12.154322 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:12.154143 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:12.155332 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:12.155310 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:04:12.946792 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:12.946487 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:13.163229 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:13.163186 4926 generic.go:356] "Generic (PLEG): container finished" podID="0e2c013c-ba60-4b4d-a37b-7656c6e228b5" containerID="35102a788c4924f28a8bffdd5801a4c60e1677d2dc18f8138cf1b40392ffd4d8" exitCode=0 Oct 30 21:04:13.163664 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:13.163312 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8" event={"ID":"0e2c013c-ba60-4b4d-a37b-7656c6e228b5","Type":"ContainerDied","Data":"35102a788c4924f28a8bffdd5801a4c60e1677d2dc18f8138cf1b40392ffd4d8"} Oct 30 21:04:13.172436 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:13.172406 4926 generic.go:356] "Generic (PLEG): container finished" podID="8ebb7012-3a31-42e0-9bc1-21a72e2a8140" containerID="b088883c17276465f7bbc20c440177f44d533e61ac37d5e3eafdf100f1b97bcf" exitCode=0 Oct 30 21:04:13.172578 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:13.172550 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj" event={"ID":"8ebb7012-3a31-42e0-9bc1-21a72e2a8140","Type":"ContainerDied","Data":"b088883c17276465f7bbc20c440177f44d533e61ac37d5e3eafdf100f1b97bcf"} Oct 30 21:04:13.177338 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:13.176481 4926 generic.go:356] "Generic (PLEG): container finished" podID="ee4ef697-79cd-452e-be1c-20a8a92d51c8" containerID="0315c5ed8dfcc35bc3e7057c6fbf97101179c75f117a97ddb5655f4e8ef8df85" exitCode=0 Oct 30 21:04:13.177338 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:13.176582 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz" event={"ID":"ee4ef697-79cd-452e-be1c-20a8a92d51c8","Type":"ContainerDied","Data":"0315c5ed8dfcc35bc3e7057c6fbf97101179c75f117a97ddb5655f4e8ef8df85"} Oct 30 21:04:13.180986 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:13.180962 4926 generic.go:356] "Generic (PLEG): container finished" podID="52e53d48-9908-477d-98f5-af9739c6b346" containerID="e27f858b63807acfaf919cb3f09984c961a45cb006091cd39881142df34a8ca7" exitCode=0 Oct 30 21:04:13.181917 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:13.181897 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc" event={"ID":"52e53d48-9908-477d-98f5-af9739c6b346","Type":"ContainerDied","Data":"e27f858b63807acfaf919cb3f09984c961a45cb006091cd39881142df34a8ca7"} Oct 30 21:04:13.182679 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:13.182398 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:13.183749 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:13.183696 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b" Oct 30 21:04:13.893348 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:04:13.893313 4926 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d2fe4e_4aff_498f_b177_75668fcac436.slice/crio-conmon-3ccd57c5e0253945838bd26bef16feb391b9d18c58e505c3d9aaf0341ce08440.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:04:14.195267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:14.195237 4926 generic.go:356] "Generic (PLEG): container finished" podID="52e53d48-9908-477d-98f5-af9739c6b346" containerID="3ab2783eda7439aa63003f86dc22437d30ea9211f10e7ff7c63b83caa7761181" exitCode=0 Oct 30 21:04:14.195607 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:14.195537 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc" event={"ID":"52e53d48-9908-477d-98f5-af9739c6b346","Type":"ContainerDied","Data":"3ab2783eda7439aa63003f86dc22437d30ea9211f10e7ff7c63b83caa7761181"} Oct 30 21:04:14.209079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:14.208995 4926 generic.go:356] "Generic (PLEG): container finished" podID="c7d2fe4e-4aff-498f-b177-75668fcac436" containerID="3ccd57c5e0253945838bd26bef16feb391b9d18c58e505c3d9aaf0341ce08440" exitCode=0 Oct 30 21:04:14.209372 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:14.209347 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9" event={"ID":"c7d2fe4e-4aff-498f-b177-75668fcac436","Type":"ContainerDied","Data":"3ccd57c5e0253945838bd26bef16feb391b9d18c58e505c3d9aaf0341ce08440"} Oct 30 21:04:14.212582 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:14.212548 4926 generic.go:356] "Generic (PLEG): container finished" podID="17568512-7448-46a4-ad28-fc64c94d36fa" containerID="f449882fe2fea054384617457aaf4bc4b01af8731c22d44d1d72dc7ad7ef0714" exitCode=0 Oct 30 21:04:14.213100 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:14.213049 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q" event={"ID":"17568512-7448-46a4-ad28-fc64c94d36fa","Type":"ContainerDied","Data":"f449882fe2fea054384617457aaf4bc4b01af8731c22d44d1d72dc7ad7ef0714"} Oct 30 21:04:15.220510 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:15.220466 4926 generic.go:356] "Generic (PLEG): container finished" podID="06816f46-272f-4772-9c69-f3af428dbba7" containerID="73f3130e3151409fce619841f0068ea8552acea6e12661240588d49461ed9ab1" exitCode=0 Oct 30 21:04:15.220980 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:15.220575 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q" event={"ID":"06816f46-272f-4772-9c69-f3af428dbba7","Type":"ContainerDied","Data":"73f3130e3151409fce619841f0068ea8552acea6e12661240588d49461ed9ab1"} Oct 30 21:04:15.224712 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:15.224690 4926 generic.go:356] "Generic (PLEG): container finished" podID="ee4ef697-79cd-452e-be1c-20a8a92d51c8" containerID="7a3c68f7e24e411eb977e04a7a639f5058a3995c0c9a1c60e61d405ca91b1465" exitCode=0 Oct 30 21:04:15.224799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:15.224775 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz" event={"ID":"ee4ef697-79cd-452e-be1c-20a8a92d51c8","Type":"ContainerDied","Data":"7a3c68f7e24e411eb977e04a7a639f5058a3995c0c9a1c60e61d405ca91b1465"} Oct 30 21:04:15.229262 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:15.229226 4926 generic.go:356] "Generic (PLEG): container finished" podID="0e2c013c-ba60-4b4d-a37b-7656c6e228b5" containerID="7f31dd5270685833292cc33ed643812b3b1bb4f81834abc447ed786f7b47b3d3" exitCode=0 Oct 30 21:04:15.229645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:15.229582 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8" event={"ID":"0e2c013c-ba60-4b4d-a37b-7656c6e228b5","Type":"ContainerDied","Data":"7f31dd5270685833292cc33ed643812b3b1bb4f81834abc447ed786f7b47b3d3"} Oct 30 21:04:15.232063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:15.232022 4926 generic.go:356] "Generic (PLEG): container finished" podID="8ebb7012-3a31-42e0-9bc1-21a72e2a8140" containerID="471f2658143298321c495c81c07349c54f0641ab01c4696b4e15a7d4cdade55b" exitCode=0 Oct 30 21:04:15.232158 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:15.232128 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj" event={"ID":"8ebb7012-3a31-42e0-9bc1-21a72e2a8140","Type":"ContainerDied","Data":"471f2658143298321c495c81c07349c54f0641ab01c4696b4e15a7d4cdade55b"} Oct 30 21:04:15.234808 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:15.234765 4926 generic.go:356] "Generic (PLEG): container finished" podID="17568512-7448-46a4-ad28-fc64c94d36fa" containerID="8b8a20625829531dcf1546c0804de93ad0a54e4e36fb10a5805f8d662cb16105" exitCode=0 Oct 30 21:04:15.234912 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:15.234892 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q" event={"ID":"17568512-7448-46a4-ad28-fc64c94d36fa","Type":"ContainerDied","Data":"8b8a20625829531dcf1546c0804de93ad0a54e4e36fb10a5805f8d662cb16105"} Oct 30 21:04:22.313020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:22.312975 4926 generic.go:356] "Generic (PLEG): container finished" podID="06816f46-272f-4772-9c69-f3af428dbba7" containerID="f19da87fc433db128aa01ac33d417f0d350f29b76eb0fa8f5707527f12c189bb" exitCode=0 Oct 30 21:04:22.313511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:22.313070 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q" event={"ID":"06816f46-272f-4772-9c69-f3af428dbba7","Type":"ContainerDied","Data":"f19da87fc433db128aa01ac33d417f0d350f29b76eb0fa8f5707527f12c189bb"} Oct 30 21:04:25.356185 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:25.356087 4926 generic.go:356] "Generic (PLEG): container finished" podID="ee4ef697-79cd-452e-be1c-20a8a92d51c8" containerID="c047cdb1b85265af7e99864eef317f960139b9dd7bba9a3e14176b0b8caddac2" exitCode=0 Oct 30 21:04:25.356185 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:25.356143 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz" event={"ID":"ee4ef697-79cd-452e-be1c-20a8a92d51c8","Type":"ContainerDied","Data":"c047cdb1b85265af7e99864eef317f960139b9dd7bba9a3e14176b0b8caddac2"} Oct 30 21:04:25.364533 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:25.364504 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8" event={"ID":"0e2c013c-ba60-4b4d-a37b-7656c6e228b5","Type":"ContainerStarted","Data":"76b63331f860774a4615a9ccb2233864d38b2f2367dc9c8d474df756c086df81"} Oct 30 21:04:25.368060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:25.368015 4926 generic.go:356] "Generic (PLEG): container finished" podID="567c5bbf-74c2-4d88-b460-53c5ca846f70" containerID="3ae7d6b33d305552ae6230a736336668e7c11c0e447ed88ee296b26e39e54538" exitCode=0 Oct 30 21:04:25.368202 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:25.368178 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk" event={"ID":"567c5bbf-74c2-4d88-b460-53c5ca846f70","Type":"ContainerDied","Data":"3ae7d6b33d305552ae6230a736336668e7c11c0e447ed88ee296b26e39e54538"} Oct 30 21:04:25.374724 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:25.374702 4926 generic.go:356] "Generic (PLEG): container finished" podID="8ebb7012-3a31-42e0-9bc1-21a72e2a8140" containerID="12b9b118ca8844e06da3a783e76d5c7ecdb43bbf4c6ce5ce9f0a1f566206064a" exitCode=0 Oct 30 21:04:25.374829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:25.374776 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj" event={"ID":"8ebb7012-3a31-42e0-9bc1-21a72e2a8140","Type":"ContainerDied","Data":"12b9b118ca8844e06da3a783e76d5c7ecdb43bbf4c6ce5ce9f0a1f566206064a"} Oct 30 21:04:25.378404 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:25.378382 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q" event={"ID":"17568512-7448-46a4-ad28-fc64c94d36fa","Type":"ContainerStarted","Data":"327900ae777862c6a2c039e345e08038f28dfe2fc41726d06a16bca983b7b7c2"} Oct 30 21:04:25.477949 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:25.477881 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8" podStartSLOduration=3.994294006 podStartE2EDuration="25.47786276s" totalImagesPullingTime="21.483568754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:25.475310381 +0000 UTC m=+252.596384127" watchObservedRunningTime="2026-10-30 21:04:25.47786276 +0000 UTC m=+252.598936502" Oct 30 21:04:26.387823 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.387785 4926 generic.go:356] "Generic (PLEG): container finished" podID="0e2c013c-ba60-4b4d-a37b-7656c6e228b5" containerID="76b63331f860774a4615a9ccb2233864d38b2f2367dc9c8d474df756c086df81" exitCode=0 Oct 30 21:04:26.388332 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.387920 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8" event={"ID":"0e2c013c-ba60-4b4d-a37b-7656c6e228b5","Type":"ContainerDied","Data":"76b63331f860774a4615a9ccb2233864d38b2f2367dc9c8d474df756c086df81"} Oct 30 21:04:26.391067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.390840 4926 generic.go:356] "Generic (PLEG): container finished" podID="17568512-7448-46a4-ad28-fc64c94d36fa" containerID="327900ae777862c6a2c039e345e08038f28dfe2fc41726d06a16bca983b7b7c2" exitCode=0 Oct 30 21:04:26.391589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.391558 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q" event={"ID":"17568512-7448-46a4-ad28-fc64c94d36fa","Type":"ContainerDied","Data":"327900ae777862c6a2c039e345e08038f28dfe2fc41726d06a16bca983b7b7c2"} Oct 30 21:04:26.589086 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.587008 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q" Oct 30 21:04:26.594870 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.594838 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj" Oct 30 21:04:26.665267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.665243 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ebb7012-3a31-42e0-9bc1-21a72e2a8140-kube-api-access-5nrwl\" (UniqueName: \"kubernetes.io/projected/8ebb7012-3a31-42e0-9bc1-21a72e2a8140-kube-api-access-5nrwl\") pod \"8ebb7012-3a31-42e0-9bc1-21a72e2a8140\" (UID: \"8ebb7012-3a31-42e0-9bc1-21a72e2a8140\") " Oct 30 21:04:26.665384 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.665312 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17568512-7448-46a4-ad28-fc64c94d36fa-bundle\" (UniqueName: \"kubernetes.io/empty-dir/17568512-7448-46a4-ad28-fc64c94d36fa-bundle\") pod \"17568512-7448-46a4-ad28-fc64c94d36fa\" (UID: \"17568512-7448-46a4-ad28-fc64c94d36fa\") " Oct 30 21:04:26.665384 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.665370 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17568512-7448-46a4-ad28-fc64c94d36fa-kube-api-access-n975b\" (UniqueName: \"kubernetes.io/projected/17568512-7448-46a4-ad28-fc64c94d36fa-kube-api-access-n975b\") pod \"17568512-7448-46a4-ad28-fc64c94d36fa\" (UID: \"17568512-7448-46a4-ad28-fc64c94d36fa\") " Oct 30 21:04:26.665492 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.665412 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ebb7012-3a31-42e0-9bc1-21a72e2a8140-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ebb7012-3a31-42e0-9bc1-21a72e2a8140-bundle\") pod \"8ebb7012-3a31-42e0-9bc1-21a72e2a8140\" (UID: \"8ebb7012-3a31-42e0-9bc1-21a72e2a8140\") " Oct 30 21:04:26.665492 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.665454 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17568512-7448-46a4-ad28-fc64c94d36fa-util\" (UniqueName: \"kubernetes.io/empty-dir/17568512-7448-46a4-ad28-fc64c94d36fa-util\") pod \"17568512-7448-46a4-ad28-fc64c94d36fa\" (UID: \"17568512-7448-46a4-ad28-fc64c94d36fa\") " Oct 30 21:04:26.665492 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.665483 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ebb7012-3a31-42e0-9bc1-21a72e2a8140-util\" (UniqueName: \"kubernetes.io/empty-dir/8ebb7012-3a31-42e0-9bc1-21a72e2a8140-util\") pod \"8ebb7012-3a31-42e0-9bc1-21a72e2a8140\" (UID: \"8ebb7012-3a31-42e0-9bc1-21a72e2a8140\") " Oct 30 21:04:26.671376 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.668979 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17568512-7448-46a4-ad28-fc64c94d36fa-bundle" pod "17568512-7448-46a4-ad28-fc64c94d36fa" (UID: "17568512-7448-46a4-ad28-fc64c94d36fa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:26.671376 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.670565 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ebb7012-3a31-42e0-9bc1-21a72e2a8140-kube-api-access-5nrwl" pod "8ebb7012-3a31-42e0-9bc1-21a72e2a8140" (UID: "8ebb7012-3a31-42e0-9bc1-21a72e2a8140"). InnerVolumeSpecName "kube-api-access-5nrwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:04:26.671376 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.671342 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ebb7012-3a31-42e0-9bc1-21a72e2a8140-bundle" pod "8ebb7012-3a31-42e0-9bc1-21a72e2a8140" (UID: "8ebb7012-3a31-42e0-9bc1-21a72e2a8140"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:26.672584 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.672550 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17568512-7448-46a4-ad28-fc64c94d36fa-kube-api-access-n975b" pod "17568512-7448-46a4-ad28-fc64c94d36fa" (UID: "17568512-7448-46a4-ad28-fc64c94d36fa"). InnerVolumeSpecName "kube-api-access-n975b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:04:26.673647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.673567 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ebb7012-3a31-42e0-9bc1-21a72e2a8140-util" pod "8ebb7012-3a31-42e0-9bc1-21a72e2a8140" (UID: "8ebb7012-3a31-42e0-9bc1-21a72e2a8140"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:26.674013 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.673990 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17568512-7448-46a4-ad28-fc64c94d36fa-util" pod "17568512-7448-46a4-ad28-fc64c94d36fa" (UID: "17568512-7448-46a4-ad28-fc64c94d36fa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:26.697882 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.697568 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk" Oct 30 21:04:26.701378 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.701353 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz" Oct 30 21:04:26.767752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.767722 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/567c5bbf-74c2-4d88-b460-53c5ca846f70-util\" (UniqueName: \"kubernetes.io/empty-dir/567c5bbf-74c2-4d88-b460-53c5ca846f70-util\") pod \"567c5bbf-74c2-4d88-b460-53c5ca846f70\" (UID: \"567c5bbf-74c2-4d88-b460-53c5ca846f70\") " Oct 30 21:04:26.767934 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.767797 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4ef697-79cd-452e-be1c-20a8a92d51c8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee4ef697-79cd-452e-be1c-20a8a92d51c8-bundle\") pod \"ee4ef697-79cd-452e-be1c-20a8a92d51c8\" (UID: \"ee4ef697-79cd-452e-be1c-20a8a92d51c8\") " Oct 30 21:04:26.767934 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.767839 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/567c5bbf-74c2-4d88-b460-53c5ca846f70-bundle\" (UniqueName: \"kubernetes.io/empty-dir/567c5bbf-74c2-4d88-b460-53c5ca846f70-bundle\") pod \"567c5bbf-74c2-4d88-b460-53c5ca846f70\" (UID: \"567c5bbf-74c2-4d88-b460-53c5ca846f70\") " Oct 30 21:04:26.767934 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.767921 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee4ef697-79cd-452e-be1c-20a8a92d51c8-kube-api-access-wx2rw\" (UniqueName: \"kubernetes.io/projected/ee4ef697-79cd-452e-be1c-20a8a92d51c8-kube-api-access-wx2rw\") pod \"ee4ef697-79cd-452e-be1c-20a8a92d51c8\" (UID: \"ee4ef697-79cd-452e-be1c-20a8a92d51c8\") " Oct 30 21:04:26.768081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.767958 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/567c5bbf-74c2-4d88-b460-53c5ca846f70-kube-api-access-rwkqg\" (UniqueName: \"kubernetes.io/projected/567c5bbf-74c2-4d88-b460-53c5ca846f70-kube-api-access-rwkqg\") pod \"567c5bbf-74c2-4d88-b460-53c5ca846f70\" (UID: \"567c5bbf-74c2-4d88-b460-53c5ca846f70\") " Oct 30 21:04:26.768081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.767988 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee4ef697-79cd-452e-be1c-20a8a92d51c8-util\" (UniqueName: \"kubernetes.io/empty-dir/ee4ef697-79cd-452e-be1c-20a8a92d51c8-util\") pod \"ee4ef697-79cd-452e-be1c-20a8a92d51c8\" (UID: \"ee4ef697-79cd-452e-be1c-20a8a92d51c8\") " Oct 30 21:04:26.768263 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.768245 4926 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/17568512-7448-46a4-ad28-fc64c94d36fa-bundle\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:26.768329 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.768270 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n975b\" (UniqueName: \"kubernetes.io/projected/17568512-7448-46a4-ad28-fc64c94d36fa-kube-api-access-n975b\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:26.768329 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.768286 4926 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8ebb7012-3a31-42e0-9bc1-21a72e2a8140-bundle\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:26.768329 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.768300 4926 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/17568512-7448-46a4-ad28-fc64c94d36fa-util\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:26.768329 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.768313 4926 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8ebb7012-3a31-42e0-9bc1-21a72e2a8140-util\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:26.768329 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.768326 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nrwl\" (UniqueName: \"kubernetes.io/projected/8ebb7012-3a31-42e0-9bc1-21a72e2a8140-kube-api-access-5nrwl\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:26.768806 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.768779 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4ef697-79cd-452e-be1c-20a8a92d51c8-bundle" pod "ee4ef697-79cd-452e-be1c-20a8a92d51c8" (UID: "ee4ef697-79cd-452e-be1c-20a8a92d51c8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:26.768908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.768875 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567c5bbf-74c2-4d88-b460-53c5ca846f70-bundle" pod "567c5bbf-74c2-4d88-b460-53c5ca846f70" (UID: "567c5bbf-74c2-4d88-b460-53c5ca846f70"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:26.771208 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.771170 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/567c5bbf-74c2-4d88-b460-53c5ca846f70-kube-api-access-rwkqg" pod "567c5bbf-74c2-4d88-b460-53c5ca846f70" (UID: "567c5bbf-74c2-4d88-b460-53c5ca846f70"). InnerVolumeSpecName "kube-api-access-rwkqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:04:26.773070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.773004 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee4ef697-79cd-452e-be1c-20a8a92d51c8-kube-api-access-wx2rw" pod "ee4ef697-79cd-452e-be1c-20a8a92d51c8" (UID: "ee4ef697-79cd-452e-be1c-20a8a92d51c8"). InnerVolumeSpecName "kube-api-access-wx2rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:04:26.773993 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.773971 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/567c5bbf-74c2-4d88-b460-53c5ca846f70-util" pod "567c5bbf-74c2-4d88-b460-53c5ca846f70" (UID: "567c5bbf-74c2-4d88-b460-53c5ca846f70"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:26.774915 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.774887 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee4ef697-79cd-452e-be1c-20a8a92d51c8-util" pod "ee4ef697-79cd-452e-be1c-20a8a92d51c8" (UID: "ee4ef697-79cd-452e-be1c-20a8a92d51c8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:26.869394 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.869337 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx2rw\" (UniqueName: \"kubernetes.io/projected/ee4ef697-79cd-452e-be1c-20a8a92d51c8-kube-api-access-wx2rw\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:26.869394 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.869379 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwkqg\" (UniqueName: \"kubernetes.io/projected/567c5bbf-74c2-4d88-b460-53c5ca846f70-kube-api-access-rwkqg\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:26.869629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.869413 4926 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee4ef697-79cd-452e-be1c-20a8a92d51c8-util\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:26.869629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.869427 4926 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/567c5bbf-74c2-4d88-b460-53c5ca846f70-util\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:26.869629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.869438 4926 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee4ef697-79cd-452e-be1c-20a8a92d51c8-bundle\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:26.869629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:26.869450 4926 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/567c5bbf-74c2-4d88-b460-53c5ca846f70-bundle\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:27.400712 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:27.400623 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz" Oct 30 21:04:27.401249 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:27.401106 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cdp4sz" event={"ID":"ee4ef697-79cd-452e-be1c-20a8a92d51c8","Type":"ContainerDied","Data":"c356061ab35788cd68fe0e97cd26095ceeeb7b243d750be62c7ecae2b5ad5931"} Oct 30 21:04:27.401249 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:27.401159 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c356061ab35788cd68fe0e97cd26095ceeeb7b243d750be62c7ecae2b5ad5931" Oct 30 21:04:27.403828 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:27.403726 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk" event={"ID":"567c5bbf-74c2-4d88-b460-53c5ca846f70","Type":"ContainerDied","Data":"118f4e42dacabcbbd959345070f09b6b3f3f491ef3128a87a119c71c5c208958"} Oct 30 21:04:27.403828 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:27.403754 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kr4dk" Oct 30 21:04:27.403828 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:27.403776 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="118f4e42dacabcbbd959345070f09b6b3f3f491ef3128a87a119c71c5c208958" Oct 30 21:04:27.406571 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:27.406507 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj" event={"ID":"8ebb7012-3a31-42e0-9bc1-21a72e2a8140","Type":"ContainerDied","Data":"e088edc66aee95c94807d6c2a83877c68be1e6fef0b973adf06a458072759882"} Oct 30 21:04:27.406571 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:27.406545 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e088edc66aee95c94807d6c2a83877c68be1e6fef0b973adf06a458072759882" Oct 30 21:04:27.406748 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:27.406611 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69tsjj" Oct 30 21:04:27.411206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:27.410466 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q" event={"ID":"17568512-7448-46a4-ad28-fc64c94d36fa","Type":"ContainerDied","Data":"6051ea6b8c2064f02d94884e4cbbb6a38fd4e47a1659a6902a7c10c7e9b73035"} Oct 30 21:04:27.411206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:27.410494 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be712jf2q" Oct 30 21:04:27.411206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:27.410505 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6051ea6b8c2064f02d94884e4cbbb6a38fd4e47a1659a6902a7c10c7e9b73035" Oct 30 21:04:27.560827 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:27.560799 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8" Oct 30 21:04:27.677786 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:27.677747 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e2c013c-ba60-4b4d-a37b-7656c6e228b5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e2c013c-ba60-4b4d-a37b-7656c6e228b5-bundle\") pod \"0e2c013c-ba60-4b4d-a37b-7656c6e228b5\" (UID: \"0e2c013c-ba60-4b4d-a37b-7656c6e228b5\") " Oct 30 21:04:27.677967 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:27.677813 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e2c013c-ba60-4b4d-a37b-7656c6e228b5-kube-api-access-hvf6k\" (UniqueName: \"kubernetes.io/projected/0e2c013c-ba60-4b4d-a37b-7656c6e228b5-kube-api-access-hvf6k\") pod \"0e2c013c-ba60-4b4d-a37b-7656c6e228b5\" (UID: \"0e2c013c-ba60-4b4d-a37b-7656c6e228b5\") " Oct 30 21:04:27.677967 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:27.677848 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e2c013c-ba60-4b4d-a37b-7656c6e228b5-util\" (UniqueName: \"kubernetes.io/empty-dir/0e2c013c-ba60-4b4d-a37b-7656c6e228b5-util\") pod \"0e2c013c-ba60-4b4d-a37b-7656c6e228b5\" (UID: \"0e2c013c-ba60-4b4d-a37b-7656c6e228b5\") " Oct 30 21:04:27.678778 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:27.678734 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e2c013c-ba60-4b4d-a37b-7656c6e228b5-bundle" pod "0e2c013c-ba60-4b4d-a37b-7656c6e228b5" (UID: "0e2c013c-ba60-4b4d-a37b-7656c6e228b5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:27.680575 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:27.680538 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e2c013c-ba60-4b4d-a37b-7656c6e228b5-kube-api-access-hvf6k" pod "0e2c013c-ba60-4b4d-a37b-7656c6e228b5" (UID: "0e2c013c-ba60-4b4d-a37b-7656c6e228b5"). InnerVolumeSpecName "kube-api-access-hvf6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:04:27.683762 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:27.683737 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e2c013c-ba60-4b4d-a37b-7656c6e228b5-util" pod "0e2c013c-ba60-4b4d-a37b-7656c6e228b5" (UID: "0e2c013c-ba60-4b4d-a37b-7656c6e228b5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:27.779204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:27.779180 4926 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0e2c013c-ba60-4b4d-a37b-7656c6e228b5-bundle\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:27.779300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:27.779207 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvf6k\" (UniqueName: \"kubernetes.io/projected/0e2c013c-ba60-4b4d-a37b-7656c6e228b5-kube-api-access-hvf6k\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:27.779300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:27.779221 4926 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0e2c013c-ba60-4b4d-a37b-7656c6e228b5-util\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:28.418010 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:28.417972 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q" event={"ID":"06816f46-272f-4772-9c69-f3af428dbba7","Type":"ContainerStarted","Data":"35ee4ee0c6f2002bf13ef071938d93f4ed14b98227623680455320b786851b54"} Oct 30 21:04:28.426892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:28.426859 4926 generic.go:356] "Generic (PLEG): container finished" podID="52e53d48-9908-477d-98f5-af9739c6b346" containerID="dedfe187b3dd686d43a758c4a91121939bb1d43b629d70e1f907f24252273342" exitCode=0 Oct 30 21:04:28.427099 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:28.426949 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc" event={"ID":"52e53d48-9908-477d-98f5-af9739c6b346","Type":"ContainerDied","Data":"dedfe187b3dd686d43a758c4a91121939bb1d43b629d70e1f907f24252273342"} Oct 30 21:04:28.429172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:28.429153 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8" Oct 30 21:04:28.429172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:28.429159 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ab2nt8" event={"ID":"0e2c013c-ba60-4b4d-a37b-7656c6e228b5","Type":"ContainerDied","Data":"93be3de74511fa7efb0388cd28707311345229f6c309aa0882fed6541acf809e"} Oct 30 21:04:28.429340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:28.429199 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93be3de74511fa7efb0388cd28707311345229f6c309aa0882fed6541acf809e" Oct 30 21:04:28.431349 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:28.431326 4926 generic.go:356] "Generic (PLEG): container finished" podID="c7d2fe4e-4aff-498f-b177-75668fcac436" containerID="3564aa2c4ed0d2474f6103543b6249ad0f86c96224614dcf8948ee54c6dc7285" exitCode=0 Oct 30 21:04:28.431471 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:28.431395 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9" event={"ID":"c7d2fe4e-4aff-498f-b177-75668fcac436","Type":"ContainerDied","Data":"3564aa2c4ed0d2474f6103543b6249ad0f86c96224614dcf8948ee54c6dc7285"} Oct 30 21:04:28.457827 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:28.457773 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q" podStartSLOduration=3.00265354 podStartE2EDuration="29.457758324s" totalImagesPullingTime="26.455104784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:03:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:28.456052828 +0000 UTC m=+255.577126568" watchObservedRunningTime="2026-10-30 21:04:28.457758324 +0000 UTC m=+255.578832070" Oct 30 21:04:29.438627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:29.438591 4926 generic.go:356] "Generic (PLEG): container finished" podID="06816f46-272f-4772-9c69-f3af428dbba7" containerID="35ee4ee0c6f2002bf13ef071938d93f4ed14b98227623680455320b786851b54" exitCode=0 Oct 30 21:04:29.439063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:29.438678 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q" event={"ID":"06816f46-272f-4772-9c69-f3af428dbba7","Type":"ContainerDied","Data":"35ee4ee0c6f2002bf13ef071938d93f4ed14b98227623680455320b786851b54"} Oct 30 21:04:29.573647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:29.573620 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9" Oct 30 21:04:29.601219 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:29.601192 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc" Oct 30 21:04:29.694731 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:29.694655 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7d2fe4e-4aff-498f-b177-75668fcac436-kube-api-access-94xr4\" (UniqueName: \"kubernetes.io/projected/c7d2fe4e-4aff-498f-b177-75668fcac436-kube-api-access-94xr4\") pod \"c7d2fe4e-4aff-498f-b177-75668fcac436\" (UID: \"c7d2fe4e-4aff-498f-b177-75668fcac436\") " Oct 30 21:04:29.694731 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:29.694693 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7d2fe4e-4aff-498f-b177-75668fcac436-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7d2fe4e-4aff-498f-b177-75668fcac436-bundle\") pod \"c7d2fe4e-4aff-498f-b177-75668fcac436\" (UID: \"c7d2fe4e-4aff-498f-b177-75668fcac436\") " Oct 30 21:04:29.694731 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:29.694732 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e53d48-9908-477d-98f5-af9739c6b346-util\" (UniqueName: \"kubernetes.io/empty-dir/52e53d48-9908-477d-98f5-af9739c6b346-util\") pod \"52e53d48-9908-477d-98f5-af9739c6b346\" (UID: \"52e53d48-9908-477d-98f5-af9739c6b346\") " Oct 30 21:04:29.695011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:29.694771 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52e53d48-9908-477d-98f5-af9739c6b346-kube-api-access-ljnxm\" (UniqueName: \"kubernetes.io/projected/52e53d48-9908-477d-98f5-af9739c6b346-kube-api-access-ljnxm\") pod \"52e53d48-9908-477d-98f5-af9739c6b346\" (UID: \"52e53d48-9908-477d-98f5-af9739c6b346\") " Oct 30 21:04:29.695011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:29.694794 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7d2fe4e-4aff-498f-b177-75668fcac436-util\" (UniqueName: \"kubernetes.io/empty-dir/c7d2fe4e-4aff-498f-b177-75668fcac436-util\") pod \"c7d2fe4e-4aff-498f-b177-75668fcac436\" (UID: \"c7d2fe4e-4aff-498f-b177-75668fcac436\") " Oct 30 21:04:29.695011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:29.694845 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e53d48-9908-477d-98f5-af9739c6b346-bundle\" (UniqueName: \"kubernetes.io/empty-dir/52e53d48-9908-477d-98f5-af9739c6b346-bundle\") pod \"52e53d48-9908-477d-98f5-af9739c6b346\" (UID: \"52e53d48-9908-477d-98f5-af9739c6b346\") " Oct 30 21:04:29.695237 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:29.695215 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d2fe4e-4aff-498f-b177-75668fcac436-bundle" pod "c7d2fe4e-4aff-498f-b177-75668fcac436" (UID: "c7d2fe4e-4aff-498f-b177-75668fcac436"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:29.695457 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:29.695426 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e53d48-9908-477d-98f5-af9739c6b346-bundle" pod "52e53d48-9908-477d-98f5-af9739c6b346" (UID: "52e53d48-9908-477d-98f5-af9739c6b346"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:29.696987 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:29.696962 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e53d48-9908-477d-98f5-af9739c6b346-kube-api-access-ljnxm" pod "52e53d48-9908-477d-98f5-af9739c6b346" (UID: "52e53d48-9908-477d-98f5-af9739c6b346"). InnerVolumeSpecName "kube-api-access-ljnxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:04:29.697268 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:29.697215 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7d2fe4e-4aff-498f-b177-75668fcac436-kube-api-access-94xr4" pod "c7d2fe4e-4aff-498f-b177-75668fcac436" (UID: "c7d2fe4e-4aff-498f-b177-75668fcac436"). InnerVolumeSpecName "kube-api-access-94xr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:04:29.700638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:29.700607 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7d2fe4e-4aff-498f-b177-75668fcac436-util" pod "c7d2fe4e-4aff-498f-b177-75668fcac436" (UID: "c7d2fe4e-4aff-498f-b177-75668fcac436"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:29.701392 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:29.701368 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e53d48-9908-477d-98f5-af9739c6b346-util" pod "52e53d48-9908-477d-98f5-af9739c6b346" (UID: "52e53d48-9908-477d-98f5-af9739c6b346"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:29.796450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:29.796411 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljnxm\" (UniqueName: \"kubernetes.io/projected/52e53d48-9908-477d-98f5-af9739c6b346-kube-api-access-ljnxm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:29.796450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:29.796444 4926 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7d2fe4e-4aff-498f-b177-75668fcac436-util\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:29.796450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:29.796455 4926 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/52e53d48-9908-477d-98f5-af9739c6b346-bundle\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:29.796689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:29.796465 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94xr4\" (UniqueName: \"kubernetes.io/projected/c7d2fe4e-4aff-498f-b177-75668fcac436-kube-api-access-94xr4\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:29.796689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:29.796475 4926 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7d2fe4e-4aff-498f-b177-75668fcac436-bundle\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:29.796689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:29.796482 4926 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/52e53d48-9908-477d-98f5-af9739c6b346-util\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:30.444829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:30.444795 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc" Oct 30 21:04:30.444829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:30.444811 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d72gncc" event={"ID":"52e53d48-9908-477d-98f5-af9739c6b346","Type":"ContainerDied","Data":"395b3d833d4cef4fa862360284518ca713ca096c6bc2c9fac2669400a0330496"} Oct 30 21:04:30.445366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:30.444866 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="395b3d833d4cef4fa862360284518ca713ca096c6bc2c9fac2669400a0330496" Oct 30 21:04:30.446858 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:30.446825 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9" event={"ID":"c7d2fe4e-4aff-498f-b177-75668fcac436","Type":"ContainerDied","Data":"7ba1940ce68c23209797ef0728e3ba10f717b764b9fb3f22e04c3cb65425a0a7"} Oct 30 21:04:30.446981 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:30.446886 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ba1940ce68c23209797ef0728e3ba10f717b764b9fb3f22e04c3cb65425a0a7" Oct 30 21:04:30.446981 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:30.446848 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xf6n9" Oct 30 21:04:30.599317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:30.599293 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q" Oct 30 21:04:30.702939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:30.702865 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06816f46-272f-4772-9c69-f3af428dbba7-kube-api-access-swbzh\" (UniqueName: \"kubernetes.io/projected/06816f46-272f-4772-9c69-f3af428dbba7-kube-api-access-swbzh\") pod \"06816f46-272f-4772-9c69-f3af428dbba7\" (UID: \"06816f46-272f-4772-9c69-f3af428dbba7\") " Oct 30 21:04:30.703097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:30.702978 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06816f46-272f-4772-9c69-f3af428dbba7-util\" (UniqueName: \"kubernetes.io/empty-dir/06816f46-272f-4772-9c69-f3af428dbba7-util\") pod \"06816f46-272f-4772-9c69-f3af428dbba7\" (UID: \"06816f46-272f-4772-9c69-f3af428dbba7\") " Oct 30 21:04:30.703097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:30.703004 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06816f46-272f-4772-9c69-f3af428dbba7-bundle\" (UniqueName: \"kubernetes.io/empty-dir/06816f46-272f-4772-9c69-f3af428dbba7-bundle\") pod \"06816f46-272f-4772-9c69-f3af428dbba7\" (UID: \"06816f46-272f-4772-9c69-f3af428dbba7\") " Oct 30 21:04:30.703779 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:30.703747 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06816f46-272f-4772-9c69-f3af428dbba7-bundle" pod "06816f46-272f-4772-9c69-f3af428dbba7" (UID: "06816f46-272f-4772-9c69-f3af428dbba7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:30.705074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:30.705050 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06816f46-272f-4772-9c69-f3af428dbba7-kube-api-access-swbzh" pod "06816f46-272f-4772-9c69-f3af428dbba7" (UID: "06816f46-272f-4772-9c69-f3af428dbba7"). InnerVolumeSpecName "kube-api-access-swbzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:04:30.707524 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:30.707502 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06816f46-272f-4772-9c69-f3af428dbba7-util" pod "06816f46-272f-4772-9c69-f3af428dbba7" (UID: "06816f46-272f-4772-9c69-f3af428dbba7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:04:30.803719 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:30.803692 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swbzh\" (UniqueName: \"kubernetes.io/projected/06816f46-272f-4772-9c69-f3af428dbba7-kube-api-access-swbzh\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:30.803719 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:30.803715 4926 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/06816f46-272f-4772-9c69-f3af428dbba7-util\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:30.803719 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:30.803724 4926 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/06816f46-272f-4772-9c69-f3af428dbba7-bundle\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:04:31.453577 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:31.453547 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q" Oct 30 21:04:31.454027 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:31.453576 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651ccvp2q" event={"ID":"06816f46-272f-4772-9c69-f3af428dbba7","Type":"ContainerDied","Data":"54df055841ea85eb9735f9e7333f2d9914b712ac875586b227c184c575ec4ee7"} Oct 30 21:04:31.454027 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:31.453623 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54df055841ea85eb9735f9e7333f2d9914b712ac875586b227c184c575ec4ee7" Oct 30 21:04:35.131047 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.130997 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm"] Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131335 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c7d2fe4e-4aff-498f-b177-75668fcac436" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131348 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7d2fe4e-4aff-498f-b177-75668fcac436" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131360 4926 cpu_manager.go:414] "containerMap: removing container" podUID="0e2c013c-ba60-4b4d-a37b-7656c6e228b5" containerName="util" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131365 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e2c013c-ba60-4b4d-a37b-7656c6e228b5" containerName="util" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131372 4926 cpu_manager.go:414] "containerMap: removing container" podUID="06816f46-272f-4772-9c69-f3af428dbba7" containerName="util" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131377 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06816f46-272f-4772-9c69-f3af428dbba7" containerName="util" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131387 4926 cpu_manager.go:414] "containerMap: removing container" podUID="8ebb7012-3a31-42e0-9bc1-21a72e2a8140" containerName="util" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131391 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ebb7012-3a31-42e0-9bc1-21a72e2a8140" containerName="util" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131410 4926 cpu_manager.go:414] "containerMap: removing container" podUID="ee4ef697-79cd-452e-be1c-20a8a92d51c8" containerName="pull" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131416 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee4ef697-79cd-452e-be1c-20a8a92d51c8" containerName="pull" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131421 4926 cpu_manager.go:414] "containerMap: removing container" podUID="8ebb7012-3a31-42e0-9bc1-21a72e2a8140" containerName="pull" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131425 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ebb7012-3a31-42e0-9bc1-21a72e2a8140" containerName="pull" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131432 4926 cpu_manager.go:414] "containerMap: removing container" podUID="17568512-7448-46a4-ad28-fc64c94d36fa" containerName="util" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131436 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17568512-7448-46a4-ad28-fc64c94d36fa" containerName="util" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131442 4926 cpu_manager.go:414] "containerMap: removing container" podUID="17568512-7448-46a4-ad28-fc64c94d36fa" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131459 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17568512-7448-46a4-ad28-fc64c94d36fa" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131466 4926 cpu_manager.go:414] "containerMap: removing container" podUID="52e53d48-9908-477d-98f5-af9739c6b346" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131470 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52e53d48-9908-477d-98f5-af9739c6b346" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131487 4926 cpu_manager.go:414] "containerMap: removing container" podUID="52e53d48-9908-477d-98f5-af9739c6b346" containerName="pull" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131492 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52e53d48-9908-477d-98f5-af9739c6b346" containerName="pull" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131499 4926 cpu_manager.go:414] "containerMap: removing container" podUID="06816f46-272f-4772-9c69-f3af428dbba7" containerName="pull" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131504 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06816f46-272f-4772-9c69-f3af428dbba7" containerName="pull" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131510 4926 cpu_manager.go:414] "containerMap: removing container" podUID="0e2c013c-ba60-4b4d-a37b-7656c6e228b5" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131514 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e2c013c-ba60-4b4d-a37b-7656c6e228b5" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131520 4926 cpu_manager.go:414] "containerMap: removing container" podUID="06816f46-272f-4772-9c69-f3af428dbba7" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131525 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06816f46-272f-4772-9c69-f3af428dbba7" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131531 4926 cpu_manager.go:414] "containerMap: removing container" podUID="ee4ef697-79cd-452e-be1c-20a8a92d51c8" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131535 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee4ef697-79cd-452e-be1c-20a8a92d51c8" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131542 4926 cpu_manager.go:414] "containerMap: removing container" podUID="567c5bbf-74c2-4d88-b460-53c5ca846f70" containerName="pull" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131546 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="567c5bbf-74c2-4d88-b460-53c5ca846f70" containerName="pull" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131553 4926 cpu_manager.go:414] "containerMap: removing container" podUID="8ebb7012-3a31-42e0-9bc1-21a72e2a8140" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131561 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ebb7012-3a31-42e0-9bc1-21a72e2a8140" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131570 4926 cpu_manager.go:414] "containerMap: removing container" podUID="ee4ef697-79cd-452e-be1c-20a8a92d51c8" containerName="util" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131575 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee4ef697-79cd-452e-be1c-20a8a92d51c8" containerName="util" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131580 4926 cpu_manager.go:414] "containerMap: removing container" podUID="17568512-7448-46a4-ad28-fc64c94d36fa" containerName="pull" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131584 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17568512-7448-46a4-ad28-fc64c94d36fa" containerName="pull" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131591 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c7d2fe4e-4aff-498f-b177-75668fcac436" containerName="util" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131595 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7d2fe4e-4aff-498f-b177-75668fcac436" containerName="util" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131600 4926 cpu_manager.go:414] "containerMap: removing container" podUID="567c5bbf-74c2-4d88-b460-53c5ca846f70" containerName="util" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131604 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="567c5bbf-74c2-4d88-b460-53c5ca846f70" containerName="util" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131612 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c7d2fe4e-4aff-498f-b177-75668fcac436" containerName="pull" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131616 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7d2fe4e-4aff-498f-b177-75668fcac436" containerName="pull" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131622 4926 cpu_manager.go:414] "containerMap: removing container" podUID="52e53d48-9908-477d-98f5-af9739c6b346" containerName="util" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131627 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52e53d48-9908-477d-98f5-af9739c6b346" containerName="util" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131636 4926 cpu_manager.go:414] "containerMap: removing container" podUID="0e2c013c-ba60-4b4d-a37b-7656c6e228b5" containerName="pull" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131642 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e2c013c-ba60-4b4d-a37b-7656c6e228b5" containerName="pull" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131647 4926 cpu_manager.go:414] "containerMap: removing container" podUID="567c5bbf-74c2-4d88-b460-53c5ca846f70" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131652 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="567c5bbf-74c2-4d88-b460-53c5ca846f70" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131708 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ebb7012-3a31-42e0-9bc1-21a72e2a8140" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131722 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="567c5bbf-74c2-4d88-b460-53c5ca846f70" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131729 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7d2fe4e-4aff-498f-b177-75668fcac436" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131740 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="17568512-7448-46a4-ad28-fc64c94d36fa" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131748 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="52e53d48-9908-477d-98f5-af9739c6b346" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131757 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="06816f46-272f-4772-9c69-f3af428dbba7" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131767 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e2c013c-ba60-4b4d-a37b-7656c6e228b5" containerName="extract" Oct 30 21:04:35.149190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.131775 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee4ef697-79cd-452e-be1c-20a8a92d51c8" containerName="extract" Oct 30 21:04:35.153280 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.153260 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" Oct 30 21:04:35.166709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.166682 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm"] Oct 30 21:04:35.176385 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.176363 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:04:35.176385 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.176382 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-t56h2\"" Oct 30 21:04:35.178167 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.178147 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:04:35.244727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.244694 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e576e4aa-33e7-43c2-b320-169f72e4ddb7-webhook-cert\") pod \"opendatahub-operator-controller-manager-98586cfbd-s6nwm\" (UID: \"e576e4aa-33e7-43c2-b320-169f72e4ddb7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" Oct 30 21:04:35.244907 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.244749 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e576e4aa-33e7-43c2-b320-169f72e4ddb7-metrics-cert\") pod \"opendatahub-operator-controller-manager-98586cfbd-s6nwm\" (UID: \"e576e4aa-33e7-43c2-b320-169f72e4ddb7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" Oct 30 21:04:35.244907 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.244856 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vglnq\" (UniqueName: \"kubernetes.io/projected/e576e4aa-33e7-43c2-b320-169f72e4ddb7-kube-api-access-vglnq\") pod \"opendatahub-operator-controller-manager-98586cfbd-s6nwm\" (UID: \"e576e4aa-33e7-43c2-b320-169f72e4ddb7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" Oct 30 21:04:35.244978 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.244950 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e576e4aa-33e7-43c2-b320-169f72e4ddb7-manifests\") pod \"opendatahub-operator-controller-manager-98586cfbd-s6nwm\" (UID: \"e576e4aa-33e7-43c2-b320-169f72e4ddb7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" Oct 30 21:04:35.245015 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.244981 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e576e4aa-33e7-43c2-b320-169f72e4ddb7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-98586cfbd-s6nwm\" (UID: \"e576e4aa-33e7-43c2-b320-169f72e4ddb7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" Oct 30 21:04:35.245079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.245014 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e576e4aa-33e7-43c2-b320-169f72e4ddb7-tmp\") pod \"opendatahub-operator-controller-manager-98586cfbd-s6nwm\" (UID: \"e576e4aa-33e7-43c2-b320-169f72e4ddb7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" Oct 30 21:04:35.268113 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.268089 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:04:35.268341 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.268326 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 21:04:35.268411 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.268398 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:04:35.345901 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.345869 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e576e4aa-33e7-43c2-b320-169f72e4ddb7-manifests\") pod \"opendatahub-operator-controller-manager-98586cfbd-s6nwm\" (UID: \"e576e4aa-33e7-43c2-b320-169f72e4ddb7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" Oct 30 21:04:35.346014 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.345905 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e576e4aa-33e7-43c2-b320-169f72e4ddb7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-98586cfbd-s6nwm\" (UID: \"e576e4aa-33e7-43c2-b320-169f72e4ddb7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" Oct 30 21:04:35.346014 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.345927 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e576e4aa-33e7-43c2-b320-169f72e4ddb7-tmp\") pod \"opendatahub-operator-controller-manager-98586cfbd-s6nwm\" (UID: \"e576e4aa-33e7-43c2-b320-169f72e4ddb7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" Oct 30 21:04:35.346014 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.345954 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e576e4aa-33e7-43c2-b320-169f72e4ddb7-webhook-cert\") pod \"opendatahub-operator-controller-manager-98586cfbd-s6nwm\" (UID: \"e576e4aa-33e7-43c2-b320-169f72e4ddb7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" Oct 30 21:04:35.346014 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.345989 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e576e4aa-33e7-43c2-b320-169f72e4ddb7-metrics-cert\") pod \"opendatahub-operator-controller-manager-98586cfbd-s6nwm\" (UID: \"e576e4aa-33e7-43c2-b320-169f72e4ddb7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" Oct 30 21:04:35.346014 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.346007 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vglnq\" (UniqueName: \"kubernetes.io/projected/e576e4aa-33e7-43c2-b320-169f72e4ddb7-kube-api-access-vglnq\") pod \"opendatahub-operator-controller-manager-98586cfbd-s6nwm\" (UID: \"e576e4aa-33e7-43c2-b320-169f72e4ddb7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" Oct 30 21:04:35.346336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.346312 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e576e4aa-33e7-43c2-b320-169f72e4ddb7-manifests\") pod \"opendatahub-operator-controller-manager-98586cfbd-s6nwm\" (UID: \"e576e4aa-33e7-43c2-b320-169f72e4ddb7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" Oct 30 21:04:35.348436 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.348414 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e576e4aa-33e7-43c2-b320-169f72e4ddb7-tmp\") pod \"opendatahub-operator-controller-manager-98586cfbd-s6nwm\" (UID: \"e576e4aa-33e7-43c2-b320-169f72e4ddb7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" Oct 30 21:04:35.348586 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.348553 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e576e4aa-33e7-43c2-b320-169f72e4ddb7-metrics-cert\") pod \"opendatahub-operator-controller-manager-98586cfbd-s6nwm\" (UID: \"e576e4aa-33e7-43c2-b320-169f72e4ddb7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" Oct 30 21:04:35.348656 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.348623 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e576e4aa-33e7-43c2-b320-169f72e4ddb7-webhook-cert\") pod \"opendatahub-operator-controller-manager-98586cfbd-s6nwm\" (UID: \"e576e4aa-33e7-43c2-b320-169f72e4ddb7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" Oct 30 21:04:35.348656 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.348631 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e576e4aa-33e7-43c2-b320-169f72e4ddb7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-98586cfbd-s6nwm\" (UID: \"e576e4aa-33e7-43c2-b320-169f72e4ddb7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" Oct 30 21:04:35.365508 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.365486 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vglnq\" (UniqueName: \"kubernetes.io/projected/e576e4aa-33e7-43c2-b320-169f72e4ddb7-kube-api-access-vglnq\") pod \"opendatahub-operator-controller-manager-98586cfbd-s6nwm\" (UID: \"e576e4aa-33e7-43c2-b320-169f72e4ddb7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" Oct 30 21:04:35.505728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.505693 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" Oct 30 21:04:35.847707 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:35.847680 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm"] Oct 30 21:04:35.850297 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:04:35.850267 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode576e4aa_33e7_43c2_b320_169f72e4ddb7.slice/crio-08f7fbfa605de651fd9bc323ecb61d9f04163d48c2c3db2548deadae234371e1 WatchSource:0}: Error finding container 08f7fbfa605de651fd9bc323ecb61d9f04163d48c2c3db2548deadae234371e1: Status 404 returned error can't find the container with id 08f7fbfa605de651fd9bc323ecb61d9f04163d48c2c3db2548deadae234371e1 Oct 30 21:04:36.476992 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:36.476949 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" event={"ID":"e576e4aa-33e7-43c2-b320-169f72e4ddb7","Type":"ContainerStarted","Data":"08f7fbfa605de651fd9bc323ecb61d9f04163d48c2c3db2548deadae234371e1"} Oct 30 21:04:37.486772 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:04:37.486704 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:04:39.494500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:39.494425 4926 generic.go:356] "Generic (PLEG): container finished" podID="e576e4aa-33e7-43c2-b320-169f72e4ddb7" containerID="d12f10bf753c10c05539cfd7eba6020f31fcbdca7bee31fc3f6a78c2fcab3434" exitCode=0 Oct 30 21:04:39.494500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:39.494480 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" event={"ID":"e576e4aa-33e7-43c2-b320-169f72e4ddb7","Type":"ContainerDied","Data":"d12f10bf753c10c05539cfd7eba6020f31fcbdca7bee31fc3f6a78c2fcab3434"} Oct 30 21:04:40.501302 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:40.501266 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" event={"ID":"e576e4aa-33e7-43c2-b320-169f72e4ddb7","Type":"ContainerStarted","Data":"6478045678e62d443f26d1a3e22e6033d13a1252a4ae4ed1966bf84694913768"} Oct 30 21:04:40.501781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:40.501364 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" Oct 30 21:04:40.583979 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:40.583929 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" podStartSLOduration=2.039477002 podStartE2EDuration="5.583915184s" totalImagesPullingTime="3.072563553s" totalInitContainerRuntime="471.874629ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:40.572205172 +0000 UTC m=+267.693278919" watchObservedRunningTime="2026-10-30 21:04:40.583915184 +0000 UTC m=+267.704988931" Oct 30 21:04:41.570455 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:41.570423 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlfjs"] Oct 30 21:04:41.572716 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:41.572697 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlfjs" Oct 30 21:04:41.601169 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:41.601141 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:04:41.601273 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:41.601169 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 21:04:41.601334 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:41.601312 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-9mcdg\"" Oct 30 21:04:41.602908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:41.602890 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlfjs"] Oct 30 21:04:41.704777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:41.704753 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9aa12227-5e0a-4e7a-9651-30e502b66af1-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dlfjs\" (UID: \"9aa12227-5e0a-4e7a-9651-30e502b66af1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlfjs" Oct 30 21:04:41.704912 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:41.704819 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vtw6\" (UniqueName: \"kubernetes.io/projected/9aa12227-5e0a-4e7a-9651-30e502b66af1-kube-api-access-7vtw6\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dlfjs\" (UID: \"9aa12227-5e0a-4e7a-9651-30e502b66af1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlfjs" Oct 30 21:04:41.805645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:41.805613 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vtw6\" (UniqueName: \"kubernetes.io/projected/9aa12227-5e0a-4e7a-9651-30e502b66af1-kube-api-access-7vtw6\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dlfjs\" (UID: \"9aa12227-5e0a-4e7a-9651-30e502b66af1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlfjs" Oct 30 21:04:41.805793 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:41.805667 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9aa12227-5e0a-4e7a-9651-30e502b66af1-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dlfjs\" (UID: \"9aa12227-5e0a-4e7a-9651-30e502b66af1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlfjs" Oct 30 21:04:41.806022 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:41.806006 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9aa12227-5e0a-4e7a-9651-30e502b66af1-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dlfjs\" (UID: \"9aa12227-5e0a-4e7a-9651-30e502b66af1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlfjs" Oct 30 21:04:41.834147 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:41.834090 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vtw6\" (UniqueName: \"kubernetes.io/projected/9aa12227-5e0a-4e7a-9651-30e502b66af1-kube-api-access-7vtw6\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-dlfjs\" (UID: \"9aa12227-5e0a-4e7a-9651-30e502b66af1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlfjs" Oct 30 21:04:41.882403 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:41.882377 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlfjs" Oct 30 21:04:42.212220 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:42.212185 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlfjs"] Oct 30 21:04:42.215698 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:04:42.215668 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa12227_5e0a_4e7a_9651_30e502b66af1.slice/crio-cefc0975b5a860159442c6f8e6c840dd46bc39ba844a576e34fdaee67717aa4a WatchSource:0}: Error finding container cefc0975b5a860159442c6f8e6c840dd46bc39ba844a576e34fdaee67717aa4a: Status 404 returned error can't find the container with id cefc0975b5a860159442c6f8e6c840dd46bc39ba844a576e34fdaee67717aa4a Oct 30 21:04:42.511589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:42.511501 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlfjs" event={"ID":"9aa12227-5e0a-4e7a-9651-30e502b66af1","Type":"ContainerStarted","Data":"cefc0975b5a860159442c6f8e6c840dd46bc39ba844a576e34fdaee67717aa4a"} Oct 30 21:04:44.522835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:44.522800 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlfjs" event={"ID":"9aa12227-5e0a-4e7a-9651-30e502b66af1","Type":"ContainerStarted","Data":"84fed8edd1b1491e7ce63ab7caf7502708d6084d7f77d74e68f671bd794b9eb2"} Oct 30 21:04:44.603019 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:44.602960 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-dlfjs" podStartSLOduration=2.187100106 podStartE2EDuration="3.60294627s" totalImagesPullingTime="1.415846164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:44.601274536 +0000 UTC m=+271.722348283" watchObservedRunningTime="2026-10-30 21:04:44.60294627 +0000 UTC m=+271.724020012" Oct 30 21:04:46.127860 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:46.127825 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8k468"] Oct 30 21:04:46.131717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:46.131698 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8k468" Oct 30 21:04:46.150600 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:46.150577 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 21:04:46.161823 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:46.161800 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 21:04:46.162211 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:46.162191 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-wcxzz\"" Oct 30 21:04:46.163813 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:46.163795 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8k468"] Oct 30 21:04:46.245788 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:46.245751 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4tzv\" (UniqueName: \"kubernetes.io/projected/97d86e50-7f90-4650-bef4-2a4cd91c1d61-kube-api-access-q4tzv\") pod \"cert-manager-webhook-5dbbbf75cc-8k468\" (UID: \"97d86e50-7f90-4650-bef4-2a4cd91c1d61\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8k468" Oct 30 21:04:46.245973 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:46.245870 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97d86e50-7f90-4650-bef4-2a4cd91c1d61-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8k468\" (UID: \"97d86e50-7f90-4650-bef4-2a4cd91c1d61\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8k468" Oct 30 21:04:46.346419 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:46.346384 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97d86e50-7f90-4650-bef4-2a4cd91c1d61-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8k468\" (UID: \"97d86e50-7f90-4650-bef4-2a4cd91c1d61\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8k468" Oct 30 21:04:46.346625 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:46.346487 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4tzv\" (UniqueName: \"kubernetes.io/projected/97d86e50-7f90-4650-bef4-2a4cd91c1d61-kube-api-access-q4tzv\") pod \"cert-manager-webhook-5dbbbf75cc-8k468\" (UID: \"97d86e50-7f90-4650-bef4-2a4cd91c1d61\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8k468" Oct 30 21:04:46.367722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:46.367685 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4tzv\" (UniqueName: \"kubernetes.io/projected/97d86e50-7f90-4650-bef4-2a4cd91c1d61-kube-api-access-q4tzv\") pod \"cert-manager-webhook-5dbbbf75cc-8k468\" (UID: \"97d86e50-7f90-4650-bef4-2a4cd91c1d61\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8k468" Oct 30 21:04:46.370917 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:46.370889 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97d86e50-7f90-4650-bef4-2a4cd91c1d61-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-8k468\" (UID: \"97d86e50-7f90-4650-bef4-2a4cd91c1d61\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8k468" Oct 30 21:04:46.455146 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:46.455117 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8k468" Oct 30 21:04:46.750617 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:46.750588 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-8k468"] Oct 30 21:04:46.753186 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:04:46.753158 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97d86e50_7f90_4650_bef4_2a4cd91c1d61.slice/crio-384531eca1f01f335c9bb8534f05f30166cd27bf4a45d4c9172cc7a63695ada7 WatchSource:0}: Error finding container 384531eca1f01f335c9bb8534f05f30166cd27bf4a45d4c9172cc7a63695ada7: Status 404 returned error can't find the container with id 384531eca1f01f335c9bb8534f05f30166cd27bf4a45d4c9172cc7a63695ada7 Oct 30 21:04:47.539771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:47.539731 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8k468" event={"ID":"97d86e50-7f90-4650-bef4-2a4cd91c1d61","Type":"ContainerStarted","Data":"384531eca1f01f335c9bb8534f05f30166cd27bf4a45d4c9172cc7a63695ada7"} Oct 30 21:04:49.550598 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:49.550513 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8k468" event={"ID":"97d86e50-7f90-4650-bef4-2a4cd91c1d61","Type":"ContainerStarted","Data":"1b958e7b6ac628eec7b468e0a19b20edfcc9872dff63656371b0a48d1949afe4"} Oct 30 21:04:49.550968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:49.550659 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8k468" Oct 30 21:04:49.594674 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:49.594621 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8k468" podStartSLOduration=1.062391048 podStartE2EDuration="3.594606818s" totalImagesPullingTime="2.53221577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:49.594421358 +0000 UTC m=+276.715495103" watchObservedRunningTime="2026-10-30 21:04:49.594606818 +0000 UTC m=+276.715680564" Oct 30 21:04:51.503705 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:51.503670 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-qghvs"] Oct 30 21:04:51.506023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:51.506006 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qghvs" Oct 30 21:04:51.507592 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:51.507568 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-98586cfbd-s6nwm" Oct 30 21:04:51.531846 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:51.531825 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-l95kn\"" Oct 30 21:04:51.531990 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:51.531977 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 21:04:51.532087 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:51.531995 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Oct 30 21:04:51.532087 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:51.532075 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 21:04:51.533787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:51.533764 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-qghvs"] Oct 30 21:04:51.594383 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:51.594363 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gp7g\" (UniqueName: \"kubernetes.io/projected/35046001-5dfb-4df4-b0bb-602292f94643-kube-api-access-4gp7g\") pod \"dns-operator-controller-manager-8557cbdb84-qghvs\" (UID: \"35046001-5dfb-4df4-b0bb-602292f94643\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qghvs" Oct 30 21:04:51.695166 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:51.695134 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gp7g\" (UniqueName: \"kubernetes.io/projected/35046001-5dfb-4df4-b0bb-602292f94643-kube-api-access-4gp7g\") pod \"dns-operator-controller-manager-8557cbdb84-qghvs\" (UID: \"35046001-5dfb-4df4-b0bb-602292f94643\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qghvs" Oct 30 21:04:51.714444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:51.714418 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gp7g\" (UniqueName: \"kubernetes.io/projected/35046001-5dfb-4df4-b0bb-602292f94643-kube-api-access-4gp7g\") pod \"dns-operator-controller-manager-8557cbdb84-qghvs\" (UID: \"35046001-5dfb-4df4-b0bb-602292f94643\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qghvs" Oct 30 21:04:51.817373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:51.817297 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qghvs" Oct 30 21:04:52.109762 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:52.109678 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-qghvs"] Oct 30 21:04:52.112501 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:04:52.112474 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35046001_5dfb_4df4_b0bb_602292f94643.slice/crio-7440ec0b18d902b18fb0d39571254defc256d565cf43919fa8aa38e754cfd7d0 WatchSource:0}: Error finding container 7440ec0b18d902b18fb0d39571254defc256d565cf43919fa8aa38e754cfd7d0: Status 404 returned error can't find the container with id 7440ec0b18d902b18fb0d39571254defc256d565cf43919fa8aa38e754cfd7d0 Oct 30 21:04:52.565755 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:52.565719 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qghvs" event={"ID":"35046001-5dfb-4df4-b0bb-602292f94643","Type":"ContainerStarted","Data":"7440ec0b18d902b18fb0d39571254defc256d565cf43919fa8aa38e754cfd7d0"} Oct 30 21:04:55.557615 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:55.557582 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8k468" Oct 30 21:04:55.581256 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:55.581226 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qghvs" event={"ID":"35046001-5dfb-4df4-b0bb-602292f94643","Type":"ContainerStarted","Data":"e693c86a301b218f2672e3d52ed0304e581eb2b60a5e8071cdbc3658c816e6a3"} Oct 30 21:04:55.581401 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:55.581303 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qghvs" Oct 30 21:04:55.637837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:55.637788 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qghvs" podStartSLOduration=2.149310916 podStartE2EDuration="4.637773146s" totalImagesPullingTime="2.48846223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:55.63608012 +0000 UTC m=+282.757153865" watchObservedRunningTime="2026-10-30 21:04:55.637773146 +0000 UTC m=+282.758846891" Oct 30 21:04:56.264194 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:56.264145 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-jgd2h"] Oct 30 21:04:56.268442 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:56.268418 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jgd2h" Oct 30 21:04:56.300025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:56.300001 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-jwsrb\"" Oct 30 21:04:56.301979 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:56.301959 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-jgd2h"] Oct 30 21:04:56.339615 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:56.339590 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzpcb\" (UniqueName: \"kubernetes.io/projected/cd34e890-4d69-4266-8aed-c976f7c5259d-kube-api-access-fzpcb\") pod \"authorino-operator-7b7bb4bc77-jgd2h\" (UID: \"cd34e890-4d69-4266-8aed-c976f7c5259d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-jgd2h" Oct 30 21:04:56.440511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:56.440472 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzpcb\" (UniqueName: \"kubernetes.io/projected/cd34e890-4d69-4266-8aed-c976f7c5259d-kube-api-access-fzpcb\") pod \"authorino-operator-7b7bb4bc77-jgd2h\" (UID: \"cd34e890-4d69-4266-8aed-c976f7c5259d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-jgd2h" Oct 30 21:04:56.459057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:56.459001 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzpcb\" (UniqueName: \"kubernetes.io/projected/cd34e890-4d69-4266-8aed-c976f7c5259d-kube-api-access-fzpcb\") pod \"authorino-operator-7b7bb4bc77-jgd2h\" (UID: \"cd34e890-4d69-4266-8aed-c976f7c5259d\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-jgd2h" Oct 30 21:04:56.581178 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:56.581121 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jgd2h" Oct 30 21:04:56.880225 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:56.880196 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-jgd2h"] Oct 30 21:04:56.884555 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:04:56.884524 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd34e890_4d69_4266_8aed_c976f7c5259d.slice/crio-338b332a0d393d8f225eb8b25d0b51b00aefb9a4e81783d6023630747e7b3fea WatchSource:0}: Error finding container 338b332a0d393d8f225eb8b25d0b51b00aefb9a4e81783d6023630747e7b3fea: Status 404 returned error can't find the container with id 338b332a0d393d8f225eb8b25d0b51b00aefb9a4e81783d6023630747e7b3fea Oct 30 21:04:57.593404 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:57.593367 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jgd2h" event={"ID":"cd34e890-4d69-4266-8aed-c976f7c5259d","Type":"ContainerStarted","Data":"338b332a0d393d8f225eb8b25d0b51b00aefb9a4e81783d6023630747e7b3fea"} Oct 30 21:04:58.600556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:58.600481 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jgd2h" event={"ID":"cd34e890-4d69-4266-8aed-c976f7c5259d","Type":"ContainerStarted","Data":"8e57e29cf0b7345ef8e77c200a69ed7852b7ce4ea688f3cac1fa44b808d34c25"} Oct 30 21:04:58.600904 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:58.600608 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jgd2h" Oct 30 21:04:58.638507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:04:58.638459 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jgd2h" podStartSLOduration=1.225462445 podStartE2EDuration="2.638447426s" totalImagesPullingTime="1.412984981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:04:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:04:58.63787857 +0000 UTC m=+285.758952316" watchObservedRunningTime="2026-10-30 21:04:58.638447426 +0000 UTC m=+285.759521172" Oct 30 21:05:03.052107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:03.052074 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s"] Oct 30 21:05:03.055658 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:03.055640 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s" Oct 30 21:05:03.072001 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:03.071972 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rps5z\"" Oct 30 21:05:03.084910 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:03.084888 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s"] Oct 30 21:05:03.099308 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:03.099284 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f0cfe45-affa-48ba-aeea-1f2d5681a022-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jtg2s\" (UID: \"1f0cfe45-affa-48ba-aeea-1f2d5681a022\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s" Oct 30 21:05:03.099403 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:03.099322 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr8jw\" (UniqueName: \"kubernetes.io/projected/1f0cfe45-affa-48ba-aeea-1f2d5681a022-kube-api-access-tr8jw\") pod \"kuadrant-operator-controller-manager-678d87766f-jtg2s\" (UID: \"1f0cfe45-affa-48ba-aeea-1f2d5681a022\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s" Oct 30 21:05:03.200206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:03.200181 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f0cfe45-affa-48ba-aeea-1f2d5681a022-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jtg2s\" (UID: \"1f0cfe45-affa-48ba-aeea-1f2d5681a022\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s" Oct 30 21:05:03.200320 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:03.200225 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr8jw\" (UniqueName: \"kubernetes.io/projected/1f0cfe45-affa-48ba-aeea-1f2d5681a022-kube-api-access-tr8jw\") pod \"kuadrant-operator-controller-manager-678d87766f-jtg2s\" (UID: \"1f0cfe45-affa-48ba-aeea-1f2d5681a022\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s" Oct 30 21:05:03.200562 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:03.200544 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f0cfe45-affa-48ba-aeea-1f2d5681a022-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-jtg2s\" (UID: \"1f0cfe45-affa-48ba-aeea-1f2d5681a022\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s" Oct 30 21:05:03.218369 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:03.218345 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr8jw\" (UniqueName: \"kubernetes.io/projected/1f0cfe45-affa-48ba-aeea-1f2d5681a022-kube-api-access-tr8jw\") pod \"kuadrant-operator-controller-manager-678d87766f-jtg2s\" (UID: \"1f0cfe45-affa-48ba-aeea-1f2d5681a022\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s" Oct 30 21:05:03.367478 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:03.367413 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s" Oct 30 21:05:03.692739 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:03.692713 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s"] Oct 30 21:05:03.695204 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:05:03.695178 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f0cfe45_affa_48ba_aeea_1f2d5681a022.slice/crio-4a254854d50f0d278a95d7130c217a9fe89e0dba22ad27d813c3be025803211c WatchSource:0}: Error finding container 4a254854d50f0d278a95d7130c217a9fe89e0dba22ad27d813c3be025803211c: Status 404 returned error can't find the container with id 4a254854d50f0d278a95d7130c217a9fe89e0dba22ad27d813c3be025803211c Oct 30 21:05:04.637824 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:04.637780 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s" event={"ID":"1f0cfe45-affa-48ba-aeea-1f2d5681a022","Type":"ContainerStarted","Data":"4a254854d50f0d278a95d7130c217a9fe89e0dba22ad27d813c3be025803211c"} Oct 30 21:05:05.681973 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:05.681936 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-sw8wh"] Oct 30 21:05:05.685614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:05.685592 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-sw8wh" Oct 30 21:05:05.714669 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:05.714589 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-c4hfj\"" Oct 30 21:05:05.718383 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:05.718357 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-sw8wh"] Oct 30 21:05:05.725693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:05.725668 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdjft\" (UniqueName: \"kubernetes.io/projected/1fa6ac28-3aaa-4c2c-9220-a1fea9494c87-kube-api-access-vdjft\") pod \"limitador-operator-controller-manager-5778bb4449-sw8wh\" (UID: \"1fa6ac28-3aaa-4c2c-9220-a1fea9494c87\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-sw8wh" Oct 30 21:05:05.827096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:05.827025 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdjft\" (UniqueName: \"kubernetes.io/projected/1fa6ac28-3aaa-4c2c-9220-a1fea9494c87-kube-api-access-vdjft\") pod \"limitador-operator-controller-manager-5778bb4449-sw8wh\" (UID: \"1fa6ac28-3aaa-4c2c-9220-a1fea9494c87\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-sw8wh" Oct 30 21:05:05.848001 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:05.847969 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdjft\" (UniqueName: \"kubernetes.io/projected/1fa6ac28-3aaa-4c2c-9220-a1fea9494c87-kube-api-access-vdjft\") pod \"limitador-operator-controller-manager-5778bb4449-sw8wh\" (UID: \"1fa6ac28-3aaa-4c2c-9220-a1fea9494c87\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-sw8wh" Oct 30 21:05:06.000015 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:05.999928 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-sw8wh" Oct 30 21:05:06.303445 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:06.303359 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-sw8wh"] Oct 30 21:05:06.307951 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:05:06.307906 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa6ac28_3aaa_4c2c_9220_a1fea9494c87.slice/crio-da566c4df7eab0251ce2430d49aebdceb1433490871d86cb6f8fa242e8a8e134 WatchSource:0}: Error finding container da566c4df7eab0251ce2430d49aebdceb1433490871d86cb6f8fa242e8a8e134: Status 404 returned error can't find the container with id da566c4df7eab0251ce2430d49aebdceb1433490871d86cb6f8fa242e8a8e134 Oct 30 21:05:06.589917 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:06.589838 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-qghvs" Oct 30 21:05:06.651466 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:06.651423 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-sw8wh" event={"ID":"1fa6ac28-3aaa-4c2c-9220-a1fea9494c87","Type":"ContainerStarted","Data":"da566c4df7eab0251ce2430d49aebdceb1433490871d86cb6f8fa242e8a8e134"} Oct 30 21:05:07.492317 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:05:07.492258 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:05:07.658944 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:07.658909 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s" event={"ID":"1f0cfe45-affa-48ba-aeea-1f2d5681a022","Type":"ContainerStarted","Data":"17d48eb1b400e377c8f193cc7bef1b9c28f652b8f04c190ccf5282160320f2c7"} Oct 30 21:05:07.659138 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:07.659119 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s" Oct 30 21:05:07.724764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:07.724721 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s" podStartSLOduration=1.464110347 podStartE2EDuration="4.724707678s" totalImagesPullingTime="3.260597331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:07.712939181 +0000 UTC m=+294.834012926" watchObservedRunningTime="2026-10-30 21:05:07.724707678 +0000 UTC m=+294.845781424" Oct 30 21:05:08.666329 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:08.666280 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-sw8wh" event={"ID":"1fa6ac28-3aaa-4c2c-9220-a1fea9494c87","Type":"ContainerStarted","Data":"3c1b3d14ac613cec82dacb93123d39f129d9c8b046b688050f99773d751daea8"} Oct 30 21:05:08.666773 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:08.666362 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-sw8wh" Oct 30 21:05:08.712052 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:08.711983 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-sw8wh" podStartSLOduration=2.346216906 podStartE2EDuration="3.711968873s" totalImagesPullingTime="1.365751967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:08.71127051 +0000 UTC m=+295.832344256" watchObservedRunningTime="2026-10-30 21:05:08.711968873 +0000 UTC m=+295.833042618" Oct 30 21:05:08.833569 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:08.833537 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7"] Oct 30 21:05:08.837230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:08.837209 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:08.856448 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:08.856427 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-rd4lg\"" Oct 30 21:05:08.865748 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:08.865726 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7"] Oct 30 21:05:08.955124 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:08.955025 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:08.955270 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:08.955116 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:08.955270 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:08.955205 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:08.955270 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:08.955266 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:08.955411 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:08.955288 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c46l\" (UniqueName: \"kubernetes.io/projected/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-kube-api-access-8c46l\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:08.955411 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:08.955339 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:08.955411 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:08.955391 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-istio-data\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:08.955508 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:08.955426 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-istio-token\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:08.955508 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:08.955467 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:09.055840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.055808 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:09.055987 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.055849 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:09.055987 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.055865 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:09.055987 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.055883 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:09.055987 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.055939 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:09.055987 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.055961 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c46l\" (UniqueName: \"kubernetes.io/projected/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-kube-api-access-8c46l\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:09.056283 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.055993 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:09.056283 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.056021 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-istio-data\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:09.056283 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.056178 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-istio-token\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:09.056283 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.056249 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:09.056462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.056291 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:09.056620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.056592 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:09.056748 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.056686 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-istio-data\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:09.056811 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.056780 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:09.058425 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.058400 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:09.058649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.058632 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:09.074578 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.074541 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-istio-token\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:09.074941 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.074921 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c46l\" (UniqueName: \"kubernetes.io/projected/4f9713f2-a62e-49a0-b9b8-7bfb6b828f35-kube-api-access-8c46l\") pod \"data-science-gateway-data-science-gateway-class-669569fcd5lntp7\" (UID: \"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:09.149684 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.149652 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:09.481298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.481207 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7"] Oct 30 21:05:09.483824 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:05:09.483793 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9713f2_a62e_49a0_b9b8_7bfb6b828f35.slice/crio-fc720f0300357f96800bded383ba53052062df7eccbebb2e2a13334695c290c6 WatchSource:0}: Error finding container fc720f0300357f96800bded383ba53052062df7eccbebb2e2a13334695c290c6: Status 404 returned error can't find the container with id fc720f0300357f96800bded383ba53052062df7eccbebb2e2a13334695c290c6 Oct 30 21:05:09.486059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.486005 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:05:09.486164 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.486097 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:05:09.486164 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.486130 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:05:09.613445 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.613394 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-jgd2h" Oct 30 21:05:09.674164 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.674113 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" event={"ID":"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35","Type":"ContainerStarted","Data":"26931b0ca1003fca391216145b47c343503b7e84b63c24531008a6f109432e4c"} Oct 30 21:05:09.674636 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.674181 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" event={"ID":"4f9713f2-a62e-49a0-b9b8-7bfb6b828f35","Type":"ContainerStarted","Data":"fc720f0300357f96800bded383ba53052062df7eccbebb2e2a13334695c290c6"} Oct 30 21:05:09.717811 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:09.717749 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" podStartSLOduration=1.717734503 podStartE2EDuration="1.717734503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:09.71566042 +0000 UTC m=+296.836734167" watchObservedRunningTime="2026-10-30 21:05:09.717734503 +0000 UTC m=+296.838808249" Oct 30 21:05:10.150840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:10.150799 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:10.155621 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:10.155594 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:10.679275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:10.679249 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:10.680156 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:10.680134 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-669569fcd5lntp7" Oct 30 21:05:13.050384 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:13.050358 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:05:13.051090 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:13.050813 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:05:16.815543 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:16.815502 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-pz9vv"] Oct 30 21:05:16.822490 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:16.822469 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pz9vv" Oct 30 21:05:16.838202 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:16.838178 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-2bjtm\"" Oct 30 21:05:16.850133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:16.850102 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:05:16.850260 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:16.850246 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:05:16.852236 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:16.852209 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-pz9vv"] Oct 30 21:05:16.873363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:16.873336 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 21:05:16.924610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:16.924583 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3947186-e2e4-40cb-8a88-07276844926f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-pz9vv\" (UID: \"a3947186-e2e4-40cb-8a88-07276844926f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pz9vv" Oct 30 21:05:16.924610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:16.924617 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a3947186-e2e4-40cb-8a88-07276844926f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-pz9vv\" (UID: \"a3947186-e2e4-40cb-8a88-07276844926f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pz9vv" Oct 30 21:05:16.924834 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:16.924642 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3947186-e2e4-40cb-8a88-07276844926f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-pz9vv\" (UID: \"a3947186-e2e4-40cb-8a88-07276844926f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pz9vv" Oct 30 21:05:16.924913 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:16.924893 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h6l8\" (UniqueName: \"kubernetes.io/projected/a3947186-e2e4-40cb-8a88-07276844926f-kube-api-access-9h6l8\") pod \"kserve-module-controller-manager-7566cf5fc8-pz9vv\" (UID: \"a3947186-e2e4-40cb-8a88-07276844926f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pz9vv" Oct 30 21:05:17.026071 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:17.026024 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h6l8\" (UniqueName: \"kubernetes.io/projected/a3947186-e2e4-40cb-8a88-07276844926f-kube-api-access-9h6l8\") pod \"kserve-module-controller-manager-7566cf5fc8-pz9vv\" (UID: \"a3947186-e2e4-40cb-8a88-07276844926f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pz9vv" Oct 30 21:05:17.026254 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:17.026096 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3947186-e2e4-40cb-8a88-07276844926f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-pz9vv\" (UID: \"a3947186-e2e4-40cb-8a88-07276844926f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pz9vv" Oct 30 21:05:17.026254 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:17.026118 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a3947186-e2e4-40cb-8a88-07276844926f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-pz9vv\" (UID: \"a3947186-e2e4-40cb-8a88-07276844926f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pz9vv" Oct 30 21:05:17.026254 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:17.026138 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3947186-e2e4-40cb-8a88-07276844926f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-pz9vv\" (UID: \"a3947186-e2e4-40cb-8a88-07276844926f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pz9vv" Oct 30 21:05:17.026463 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:17.026447 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3947186-e2e4-40cb-8a88-07276844926f-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-pz9vv\" (UID: \"a3947186-e2e4-40cb-8a88-07276844926f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pz9vv" Oct 30 21:05:17.026539 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:17.026517 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a3947186-e2e4-40cb-8a88-07276844926f-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-pz9vv\" (UID: \"a3947186-e2e4-40cb-8a88-07276844926f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pz9vv" Oct 30 21:05:17.026911 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:17.026892 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a3947186-e2e4-40cb-8a88-07276844926f-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-pz9vv\" (UID: \"a3947186-e2e4-40cb-8a88-07276844926f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pz9vv" Oct 30 21:05:17.046892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:17.046860 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h6l8\" (UniqueName: \"kubernetes.io/projected/a3947186-e2e4-40cb-8a88-07276844926f-kube-api-access-9h6l8\") pod \"kserve-module-controller-manager-7566cf5fc8-pz9vv\" (UID: \"a3947186-e2e4-40cb-8a88-07276844926f\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pz9vv" Oct 30 21:05:17.134260 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:17.134177 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pz9vv" Oct 30 21:05:17.458023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:17.457998 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-pz9vv"] Oct 30 21:05:17.460448 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:05:17.460421 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3947186_e2e4_40cb_8a88_07276844926f.slice/crio-be5345251edef2a4431e40638f70bc020d0bef4e06062917fab219b90987cfa3 WatchSource:0}: Error finding container be5345251edef2a4431e40638f70bc020d0bef4e06062917fab219b90987cfa3: Status 404 returned error can't find the container with id be5345251edef2a4431e40638f70bc020d0bef4e06062917fab219b90987cfa3 Oct 30 21:05:17.463123 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:17.462510 4926 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:05:17.716650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:17.716563 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pz9vv" event={"ID":"a3947186-e2e4-40cb-8a88-07276844926f","Type":"ContainerStarted","Data":"be5345251edef2a4431e40638f70bc020d0bef4e06062917fab219b90987cfa3"} Oct 30 21:05:18.670189 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:18.670155 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s" Oct 30 21:05:19.677193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:19.677160 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-sw8wh" Oct 30 21:05:20.734699 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:20.734656 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pz9vv" event={"ID":"a3947186-e2e4-40cb-8a88-07276844926f","Type":"ContainerStarted","Data":"32ecb9084d2b38d0e221d0bc235795bceab2440bd55ff57b9f39fa6387b4ed4d"} Oct 30 21:05:20.735195 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:20.734724 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pz9vv" Oct 30 21:05:20.811198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:20.811150 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pz9vv" podStartSLOduration=2.5153417769999997 podStartE2EDuration="4.811136818s" totalImagesPullingTime="2.295795041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:20.80947747 +0000 UTC m=+307.930551217" watchObservedRunningTime="2026-10-30 21:05:20.811136818 +0000 UTC m=+307.932210617" Oct 30 21:05:37.483433 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:05:37.483354 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:05:41.741468 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:41.741433 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-pz9vv" Oct 30 21:05:49.245074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:49.245022 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:05:49.249216 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:49.249178 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-75ttb" Oct 30 21:05:49.275645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:49.275622 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-lmssf\"" Oct 30 21:05:49.275838 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:49.275816 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 21:05:49.277930 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:49.277905 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:05:49.305356 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:49.305328 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tw8c\" (UniqueName: \"kubernetes.io/projected/69012155-4a19-4aed-a956-28b5efd43cf1-kube-api-access-6tw8c\") pod \"limitador-limitador-7c9b87d574-75ttb\" (UID: \"69012155-4a19-4aed-a956-28b5efd43cf1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-75ttb" Oct 30 21:05:49.305493 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:49.305372 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/69012155-4a19-4aed-a956-28b5efd43cf1-config-file\") pod \"limitador-limitador-7c9b87d574-75ttb\" (UID: \"69012155-4a19-4aed-a956-28b5efd43cf1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-75ttb" Oct 30 21:05:49.330151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:49.330126 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:05:49.406576 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:49.406548 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tw8c\" (UniqueName: \"kubernetes.io/projected/69012155-4a19-4aed-a956-28b5efd43cf1-kube-api-access-6tw8c\") pod \"limitador-limitador-7c9b87d574-75ttb\" (UID: \"69012155-4a19-4aed-a956-28b5efd43cf1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-75ttb" Oct 30 21:05:49.406705 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:49.406601 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/69012155-4a19-4aed-a956-28b5efd43cf1-config-file\") pod \"limitador-limitador-7c9b87d574-75ttb\" (UID: \"69012155-4a19-4aed-a956-28b5efd43cf1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-75ttb" Oct 30 21:05:49.407457 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:49.407433 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/69012155-4a19-4aed-a956-28b5efd43cf1-config-file\") pod \"limitador-limitador-7c9b87d574-75ttb\" (UID: \"69012155-4a19-4aed-a956-28b5efd43cf1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-75ttb" Oct 30 21:05:49.425207 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:49.425173 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tw8c\" (UniqueName: \"kubernetes.io/projected/69012155-4a19-4aed-a956-28b5efd43cf1-kube-api-access-6tw8c\") pod \"limitador-limitador-7c9b87d574-75ttb\" (UID: \"69012155-4a19-4aed-a956-28b5efd43cf1\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-75ttb" Oct 30 21:05:49.561231 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:49.561149 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-75ttb" Oct 30 21:05:49.855786 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:49.855761 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:05:49.857767 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:05:49.857733 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69012155_4a19_4aed_a956_28b5efd43cf1.slice/crio-9fb634fa792f4c20480192463cb91a72b5f38599b18d01ab193c876eb53fa900 WatchSource:0}: Error finding container 9fb634fa792f4c20480192463cb91a72b5f38599b18d01ab193c876eb53fa900: Status 404 returned error can't find the container with id 9fb634fa792f4c20480192463cb91a72b5f38599b18d01ab193c876eb53fa900 Oct 30 21:05:49.879225 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:49.879197 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-75ttb" event={"ID":"69012155-4a19-4aed-a956-28b5efd43cf1","Type":"ContainerStarted","Data":"9fb634fa792f4c20480192463cb91a72b5f38599b18d01ab193c876eb53fa900"} Oct 30 21:05:51.891247 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:51.891196 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-75ttb" event={"ID":"69012155-4a19-4aed-a956-28b5efd43cf1","Type":"ContainerStarted","Data":"29c166726088801cf4e5c11a6074d882581108dc231a72f2a33ff60c71a21d60"} Oct 30 21:05:51.891693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:51.891273 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-75ttb" Oct 30 21:05:51.932100 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:05:51.932050 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-75ttb" podStartSLOduration=1.9366463729999999 podStartE2EDuration="2.932022403s" totalImagesPullingTime="995.37603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:05:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:05:51.930183445 +0000 UTC m=+339.051257192" watchObservedRunningTime="2026-10-30 21:05:51.932022403 +0000 UTC m=+339.053096150" Oct 30 21:06:02.897177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:02.897146 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-75ttb" Oct 30 21:06:07.482177 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:06:07.482138 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:06:12.693994 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:12.693955 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt"] Oct 30 21:06:12.701199 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:12.701168 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt" Oct 30 21:06:12.717470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:12.717439 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt"] Oct 30 21:06:12.736215 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:12.736188 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-dcd6m"] Oct 30 21:06:12.739740 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:12.739720 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-dcd6m" Oct 30 21:06:12.752207 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:12.752183 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 21:06:12.752301 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:12.752253 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-4lwk6\"" Oct 30 21:06:12.754520 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:12.754505 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 21:06:12.760712 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:12.760687 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-dcd6m"] Oct 30 21:06:12.773128 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:12.773109 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-dttsx\"" Oct 30 21:06:12.812721 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:12.812696 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdjgk\" (UniqueName: \"kubernetes.io/projected/0a14816b-84fb-4f2f-aa6d-b5b23beb7a6e-kube-api-access-wdjgk\") pod \"model-serving-api-75bd89fdfc-dcd6m\" (UID: \"0a14816b-84fb-4f2f-aa6d-b5b23beb7a6e\") " pod="opendatahub/model-serving-api-75bd89fdfc-dcd6m" Oct 30 21:06:12.812826 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:12.812766 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9427f36d-ab68-4dd6-af3f-1533aabc6833-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-rl2vt\" (UID: \"9427f36d-ab68-4dd6-af3f-1533aabc6833\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt" Oct 30 21:06:12.812878 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:12.812841 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl2gz\" (UniqueName: \"kubernetes.io/projected/9427f36d-ab68-4dd6-af3f-1533aabc6833-kube-api-access-gl2gz\") pod \"llmisvc-controller-manager-869d5dcc5d-rl2vt\" (UID: \"9427f36d-ab68-4dd6-af3f-1533aabc6833\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt" Oct 30 21:06:12.812930 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:12.812897 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a14816b-84fb-4f2f-aa6d-b5b23beb7a6e-tls-certs\") pod \"model-serving-api-75bd89fdfc-dcd6m\" (UID: \"0a14816b-84fb-4f2f-aa6d-b5b23beb7a6e\") " pod="opendatahub/model-serving-api-75bd89fdfc-dcd6m" Oct 30 21:06:12.913955 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:12.913923 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9427f36d-ab68-4dd6-af3f-1533aabc6833-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-rl2vt\" (UID: \"9427f36d-ab68-4dd6-af3f-1533aabc6833\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt" Oct 30 21:06:12.914161 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:12.913966 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl2gz\" (UniqueName: \"kubernetes.io/projected/9427f36d-ab68-4dd6-af3f-1533aabc6833-kube-api-access-gl2gz\") pod \"llmisvc-controller-manager-869d5dcc5d-rl2vt\" (UID: \"9427f36d-ab68-4dd6-af3f-1533aabc6833\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt" Oct 30 21:06:12.914161 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:12.914008 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a14816b-84fb-4f2f-aa6d-b5b23beb7a6e-tls-certs\") pod \"model-serving-api-75bd89fdfc-dcd6m\" (UID: \"0a14816b-84fb-4f2f-aa6d-b5b23beb7a6e\") " pod="opendatahub/model-serving-api-75bd89fdfc-dcd6m" Oct 30 21:06:12.914161 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:12.914052 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdjgk\" (UniqueName: \"kubernetes.io/projected/0a14816b-84fb-4f2f-aa6d-b5b23beb7a6e-kube-api-access-wdjgk\") pod \"model-serving-api-75bd89fdfc-dcd6m\" (UID: \"0a14816b-84fb-4f2f-aa6d-b5b23beb7a6e\") " pod="opendatahub/model-serving-api-75bd89fdfc-dcd6m" Oct 30 21:06:12.916744 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:12.916715 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a14816b-84fb-4f2f-aa6d-b5b23beb7a6e-tls-certs\") pod \"model-serving-api-75bd89fdfc-dcd6m\" (UID: \"0a14816b-84fb-4f2f-aa6d-b5b23beb7a6e\") " pod="opendatahub/model-serving-api-75bd89fdfc-dcd6m" Oct 30 21:06:12.916876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:12.916845 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9427f36d-ab68-4dd6-af3f-1533aabc6833-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-rl2vt\" (UID: \"9427f36d-ab68-4dd6-af3f-1533aabc6833\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt" Oct 30 21:06:12.933020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:12.932986 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl2gz\" (UniqueName: \"kubernetes.io/projected/9427f36d-ab68-4dd6-af3f-1533aabc6833-kube-api-access-gl2gz\") pod \"llmisvc-controller-manager-869d5dcc5d-rl2vt\" (UID: \"9427f36d-ab68-4dd6-af3f-1533aabc6833\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt" Oct 30 21:06:12.933172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:12.933057 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdjgk\" (UniqueName: \"kubernetes.io/projected/0a14816b-84fb-4f2f-aa6d-b5b23beb7a6e-kube-api-access-wdjgk\") pod \"model-serving-api-75bd89fdfc-dcd6m\" (UID: \"0a14816b-84fb-4f2f-aa6d-b5b23beb7a6e\") " pod="opendatahub/model-serving-api-75bd89fdfc-dcd6m" Oct 30 21:06:13.031140 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:13.031069 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-4lwk6\"" Oct 30 21:06:13.038641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:13.038618 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt" Oct 30 21:06:13.065231 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:13.065208 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-dttsx\"" Oct 30 21:06:13.072498 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:13.072472 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-dcd6m" Oct 30 21:06:13.334458 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:13.334309 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt"] Oct 30 21:06:13.336632 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:06:13.336602 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9427f36d_ab68_4dd6_af3f_1533aabc6833.slice/crio-602b63909cf09b546636dc17adefd2e1247974ada5bc35a1fe888276d54a1117 WatchSource:0}: Error finding container 602b63909cf09b546636dc17adefd2e1247974ada5bc35a1fe888276d54a1117: Status 404 returned error can't find the container with id 602b63909cf09b546636dc17adefd2e1247974ada5bc35a1fe888276d54a1117 Oct 30 21:06:13.372834 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:13.372812 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-dcd6m"] Oct 30 21:06:13.374466 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:06:13.374445 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a14816b_84fb_4f2f_aa6d_b5b23beb7a6e.slice/crio-532dea6a0cd28d36bf518eef35ae1a98b92d21f6ce7717f8ab1fe70fe944d476 WatchSource:0}: Error finding container 532dea6a0cd28d36bf518eef35ae1a98b92d21f6ce7717f8ab1fe70fe944d476: Status 404 returned error can't find the container with id 532dea6a0cd28d36bf518eef35ae1a98b92d21f6ce7717f8ab1fe70fe944d476 Oct 30 21:06:14.006648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:14.006600 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt" event={"ID":"9427f36d-ab68-4dd6-af3f-1533aabc6833","Type":"ContainerStarted","Data":"602b63909cf09b546636dc17adefd2e1247974ada5bc35a1fe888276d54a1117"} Oct 30 21:06:14.008722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:14.008684 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-dcd6m" event={"ID":"0a14816b-84fb-4f2f-aa6d-b5b23beb7a6e","Type":"ContainerStarted","Data":"532dea6a0cd28d36bf518eef35ae1a98b92d21f6ce7717f8ab1fe70fe944d476"} Oct 30 21:06:16.030075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:16.030006 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt" event={"ID":"9427f36d-ab68-4dd6-af3f-1533aabc6833","Type":"ContainerStarted","Data":"9e160bea6dd6e9e0e3fd47c62be09ca7732c824304d6a9ab0d92b3f71af86bac"} Oct 30 21:06:16.030554 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:16.030192 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt" Oct 30 21:06:16.031736 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:16.031712 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-dcd6m" event={"ID":"0a14816b-84fb-4f2f-aa6d-b5b23beb7a6e","Type":"ContainerStarted","Data":"508caa497e72dce2696e6cdb79a78d237f473eb00512d5c609aa842c069a9c26"} Oct 30 21:06:16.031853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:16.031838 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-dcd6m" Oct 30 21:06:16.078450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:16.078404 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt" podStartSLOduration=1.913121529 podStartE2EDuration="4.078392599s" totalImagesPullingTime="2.16527107s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:16.07720003 +0000 UTC m=+363.198273776" watchObservedRunningTime="2026-10-30 21:06:16.078392599 +0000 UTC m=+363.199466344" Oct 30 21:06:16.130184 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:16.130142 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-dcd6m" podStartSLOduration=2.335053342 podStartE2EDuration="4.130129359s" totalImagesPullingTime="1.795076017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:06:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:06:16.126734012 +0000 UTC m=+363.247807758" watchObservedRunningTime="2026-10-30 21:06:16.130129359 +0000 UTC m=+363.251203104" Oct 30 21:06:27.041812 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:27.041785 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-dcd6m" Oct 30 21:06:37.483083 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:06:37.483017 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:06:47.040128 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:06:47.040087 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt" Oct 30 21:07:07.482830 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:07:07.482793 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:07:07.857690 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:07.857608 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:07:18.351705 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:18.351672 4926 generic.go:356] "Generic (PLEG): container finished" podID="d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94" containerID="7278eeb91f6d7e18c8681619c0bfdcbd1223cc8f3b6d6f78667ff277680dedca" exitCode=0 Oct 30 21:07:18.352159 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:18.351748 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" event={"ID":"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94","Type":"ContainerDied","Data":"7278eeb91f6d7e18c8681619c0bfdcbd1223cc8f3b6d6f78667ff277680dedca"} Oct 30 21:07:18.352159 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:18.352136 4926 scope.go:122] "RemoveContainer" containerID="7278eeb91f6d7e18c8681619c0bfdcbd1223cc8f3b6d6f78667ff277680dedca" Oct 30 21:07:18.353343 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:18.353312 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:07:18.353450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:18.353382 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:07:18.353450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:18.353415 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:07:18.430409 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:18.430379 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-mq7lz"] Oct 30 21:07:18.433498 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:18.433471 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-mq7lz" Oct 30 21:07:18.453683 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:18.453657 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-mq7lz"] Oct 30 21:07:18.577353 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:18.577319 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b9f291b-8192-4185-82d3-fd195ba1c62d-cert\") pod \"llmisvc-controller-manager-6466c554d8-mq7lz\" (UID: \"0b9f291b-8192-4185-82d3-fd195ba1c62d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-mq7lz" Oct 30 21:07:18.577540 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:18.577375 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgh5k\" (UniqueName: \"kubernetes.io/projected/0b9f291b-8192-4185-82d3-fd195ba1c62d-kube-api-access-jgh5k\") pod \"llmisvc-controller-manager-6466c554d8-mq7lz\" (UID: \"0b9f291b-8192-4185-82d3-fd195ba1c62d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-mq7lz" Oct 30 21:07:18.678475 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:18.678437 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b9f291b-8192-4185-82d3-fd195ba1c62d-cert\") pod \"llmisvc-controller-manager-6466c554d8-mq7lz\" (UID: \"0b9f291b-8192-4185-82d3-fd195ba1c62d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-mq7lz" Oct 30 21:07:18.678654 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:18.678521 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgh5k\" (UniqueName: \"kubernetes.io/projected/0b9f291b-8192-4185-82d3-fd195ba1c62d-kube-api-access-jgh5k\") pod \"llmisvc-controller-manager-6466c554d8-mq7lz\" (UID: \"0b9f291b-8192-4185-82d3-fd195ba1c62d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-mq7lz" Oct 30 21:07:18.681210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:18.681183 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b9f291b-8192-4185-82d3-fd195ba1c62d-cert\") pod \"llmisvc-controller-manager-6466c554d8-mq7lz\" (UID: \"0b9f291b-8192-4185-82d3-fd195ba1c62d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-mq7lz" Oct 30 21:07:18.698919 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:18.698889 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgh5k\" (UniqueName: \"kubernetes.io/projected/0b9f291b-8192-4185-82d3-fd195ba1c62d-kube-api-access-jgh5k\") pod \"llmisvc-controller-manager-6466c554d8-mq7lz\" (UID: \"0b9f291b-8192-4185-82d3-fd195ba1c62d\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-mq7lz" Oct 30 21:07:18.750560 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:18.750523 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-mq7lz" Oct 30 21:07:19.065559 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:19.065507 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-mq7lz"] Oct 30 21:07:19.092752 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:07:19.092717 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b9f291b_8192_4185_82d3_fd195ba1c62d.slice/crio-2d7adafa3eb772d960ef7b7b4acc1084583591461df92c90efae70d4e16fea9f WatchSource:0}: Error finding container 2d7adafa3eb772d960ef7b7b4acc1084583591461df92c90efae70d4e16fea9f: Status 404 returned error can't find the container with id 2d7adafa3eb772d960ef7b7b4acc1084583591461df92c90efae70d4e16fea9f Oct 30 21:07:19.358790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:19.358701 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" event={"ID":"d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94","Type":"ContainerStarted","Data":"fcf2c6de5968709ea68bc1dee6daa20c73172abf372fb788cb22ad27e39fff31"} Oct 30 21:07:19.359954 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:19.359930 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-mq7lz" event={"ID":"0b9f291b-8192-4185-82d3-fd195ba1c62d","Type":"ContainerStarted","Data":"2d7adafa3eb772d960ef7b7b4acc1084583591461df92c90efae70d4e16fea9f"} Oct 30 21:07:19.815912 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:19.815876 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:07:19.816136 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:19.816106 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-wxstq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 21:07:19.816198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:19.816164 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" podUID="d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 21:07:20.375056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:20.371780 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-mq7lz" event={"ID":"0b9f291b-8192-4185-82d3-fd195ba1c62d","Type":"ContainerStarted","Data":"094f35cb95ea111e3cc436a5cf1df1eb8afa31a13ba17e66a8971d95118e3d52"} Oct 30 21:07:20.375056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:20.371842 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-mq7lz" Oct 30 21:07:20.413274 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:20.413225 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-mq7lz" podStartSLOduration=2.0688353839999998 podStartE2EDuration="2.413211002s" totalImagesPullingTime="344.375618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:07:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:07:20.411564676 +0000 UTC m=+427.532638421" watchObservedRunningTime="2026-10-30 21:07:20.413211002 +0000 UTC m=+427.534284748" Oct 30 21:07:20.815976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:20.815941 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-wxstq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 21:07:20.816170 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:20.815997 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" podUID="d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 21:07:21.816356 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:21.816318 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-wxstq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 21:07:21.816776 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:21.816386 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" podUID="d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 21:07:22.815965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:22.815927 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-wxstq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 21:07:22.816165 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:22.815981 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" podUID="d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 21:07:23.816599 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:23.816558 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-wxstq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" start-of-body= Oct 30 21:07:23.816998 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:23.816623 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" podUID="d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": dial tcp 10.132.0.32:15021: connect: connection refused" Oct 30 21:07:24.820131 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:24.820100 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:07:24.820521 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:24.820403 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:07:24.820986 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:24.820968 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" Oct 30 21:07:37.483271 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:07:37.483231 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:07:51.397135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:51.397108 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-mq7lz" Oct 30 21:07:51.501692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:51.501661 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt"] Oct 30 21:07:51.501927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:51.501902 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt" podUID="9427f36d-ab68-4dd6-af3f-1533aabc6833" containerName="manager" containerID="cri-o://9e160bea6dd6e9e0e3fd47c62be09ca7732c824304d6a9ab0d92b3f71af86bac" gracePeriod=30 Oct 30 21:07:51.753527 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:51.753505 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt" Oct 30 21:07:51.871695 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:51.871663 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9427f36d-ab68-4dd6-af3f-1533aabc6833-cert\" (UniqueName: \"kubernetes.io/secret/9427f36d-ab68-4dd6-af3f-1533aabc6833-cert\") pod \"9427f36d-ab68-4dd6-af3f-1533aabc6833\" (UID: \"9427f36d-ab68-4dd6-af3f-1533aabc6833\") " Oct 30 21:07:51.871859 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:51.871722 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9427f36d-ab68-4dd6-af3f-1533aabc6833-kube-api-access-gl2gz\" (UniqueName: \"kubernetes.io/projected/9427f36d-ab68-4dd6-af3f-1533aabc6833-kube-api-access-gl2gz\") pod \"9427f36d-ab68-4dd6-af3f-1533aabc6833\" (UID: \"9427f36d-ab68-4dd6-af3f-1533aabc6833\") " Oct 30 21:07:51.873894 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:51.873860 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9427f36d-ab68-4dd6-af3f-1533aabc6833-cert" pod "9427f36d-ab68-4dd6-af3f-1533aabc6833" (UID: "9427f36d-ab68-4dd6-af3f-1533aabc6833"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:07:51.873998 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:51.873897 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9427f36d-ab68-4dd6-af3f-1533aabc6833-kube-api-access-gl2gz" pod "9427f36d-ab68-4dd6-af3f-1533aabc6833" (UID: "9427f36d-ab68-4dd6-af3f-1533aabc6833"). InnerVolumeSpecName "kube-api-access-gl2gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:07:51.973400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:51.973334 4926 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9427f36d-ab68-4dd6-af3f-1533aabc6833-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:07:51.973400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:51.973358 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl2gz\" (UniqueName: \"kubernetes.io/projected/9427f36d-ab68-4dd6-af3f-1533aabc6833-kube-api-access-gl2gz\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:07:52.570760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:52.570726 4926 generic.go:356] "Generic (PLEG): container finished" podID="9427f36d-ab68-4dd6-af3f-1533aabc6833" containerID="9e160bea6dd6e9e0e3fd47c62be09ca7732c824304d6a9ab0d92b3f71af86bac" exitCode=0 Oct 30 21:07:52.571158 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:52.570771 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt" Oct 30 21:07:52.571158 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:52.570822 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt" event={"ID":"9427f36d-ab68-4dd6-af3f-1533aabc6833","Type":"ContainerDied","Data":"9e160bea6dd6e9e0e3fd47c62be09ca7732c824304d6a9ab0d92b3f71af86bac"} Oct 30 21:07:52.571158 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:52.570885 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt" event={"ID":"9427f36d-ab68-4dd6-af3f-1533aabc6833","Type":"ContainerDied","Data":"602b63909cf09b546636dc17adefd2e1247974ada5bc35a1fe888276d54a1117"} Oct 30 21:07:52.571158 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:52.570907 4926 scope.go:122] "RemoveContainer" containerID="9e160bea6dd6e9e0e3fd47c62be09ca7732c824304d6a9ab0d92b3f71af86bac" Oct 30 21:07:52.580197 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:52.580183 4926 scope.go:122] "RemoveContainer" containerID="9e160bea6dd6e9e0e3fd47c62be09ca7732c824304d6a9ab0d92b3f71af86bac" Oct 30 21:07:52.580453 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:07:52.580432 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e160bea6dd6e9e0e3fd47c62be09ca7732c824304d6a9ab0d92b3f71af86bac\": container with ID starting with 9e160bea6dd6e9e0e3fd47c62be09ca7732c824304d6a9ab0d92b3f71af86bac not found: ID does not exist" containerID="9e160bea6dd6e9e0e3fd47c62be09ca7732c824304d6a9ab0d92b3f71af86bac" Oct 30 21:07:52.580502 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:52.580461 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e160bea6dd6e9e0e3fd47c62be09ca7732c824304d6a9ab0d92b3f71af86bac"} err="failed to get container status \"9e160bea6dd6e9e0e3fd47c62be09ca7732c824304d6a9ab0d92b3f71af86bac\": rpc error: code = NotFound desc = could not find container \"9e160bea6dd6e9e0e3fd47c62be09ca7732c824304d6a9ab0d92b3f71af86bac\": container with ID starting with 9e160bea6dd6e9e0e3fd47c62be09ca7732c824304d6a9ab0d92b3f71af86bac not found: ID does not exist" Oct 30 21:07:52.628533 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:52.628505 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt"] Oct 30 21:07:52.639558 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:52.639535 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-rl2vt"] Oct 30 21:07:53.013888 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:07:53.013859 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9427f36d-ab68-4dd6-af3f-1533aabc6833" path="/var/lib/kubelet/pods/9427f36d-ab68-4dd6-af3f-1533aabc6833/volumes" Oct 30 21:08:07.482926 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:08:07.482890 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:08:15.014534 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:08:15.014502 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:08:18.591554 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:08:18.591518 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:08:23.624639 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:08:23.624604 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:08:32.536889 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:08:32.536814 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:08:37.484537 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:08:37.484493 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:09:02.411478 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.411442 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z"] Oct 30 21:09:02.411988 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.411807 4926 cpu_manager.go:414] "containerMap: removing container" podUID="9427f36d-ab68-4dd6-af3f-1533aabc6833" containerName="manager" Oct 30 21:09:02.411988 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.411818 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9427f36d-ab68-4dd6-af3f-1533aabc6833" containerName="manager" Oct 30 21:09:02.411988 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.411903 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="9427f36d-ab68-4dd6-af3f-1533aabc6833" containerName="manager" Oct 30 21:09:02.415527 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.415510 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.444457 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.444424 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z"] Oct 30 21:09:02.471230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.470401 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.471230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.470465 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99d9s\" (UniqueName: \"kubernetes.io/projected/9cbcfa4c-ad1b-4110-a069-38c79809bc98-kube-api-access-99d9s\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.471230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.470519 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.471230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.470548 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.471230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.470597 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.471230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.470626 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.471230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.470662 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.471230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.470711 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.471230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.470747 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.571287 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.571246 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.571287 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.571287 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.571491 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.571311 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.571491 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.571354 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.571491 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.571384 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.571491 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.571432 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.571491 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.571459 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99d9s\" (UniqueName: \"kubernetes.io/projected/9cbcfa4c-ad1b-4110-a069-38c79809bc98-kube-api-access-99d9s\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.571777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.571513 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.571777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.571536 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.571887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.571799 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.571887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.571843 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.571887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.571870 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.572061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.571950 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.572132 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.572106 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.573903 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.573873 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.574009 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.573986 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.590292 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.590273 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.590503 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.590481 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99d9s\" (UniqueName: \"kubernetes.io/projected/9cbcfa4c-ad1b-4110-a069-38c79809bc98-kube-api-access-99d9s\") pod \"maas-default-gateway-openshift-default-57f78bcd55-cdm7z\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:02.728582 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:02.728547 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:03.035337 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:03.035264 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z"] Oct 30 21:09:03.038375 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:09:03.038346 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cbcfa4c_ad1b_4110_a069_38c79809bc98.slice/crio-39671291116a48a1b204098de72ac7a92ac7bf74c92a12d8d1f772eef06b3794 WatchSource:0}: Error finding container 39671291116a48a1b204098de72ac7a92ac7bf74c92a12d8d1f772eef06b3794: Status 404 returned error can't find the container with id 39671291116a48a1b204098de72ac7a92ac7bf74c92a12d8d1f772eef06b3794 Oct 30 21:09:03.040243 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:03.040212 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:03.040316 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:03.040273 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:03.040316 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:03.040304 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:09:03.946833 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:03.946790 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" event={"ID":"9cbcfa4c-ad1b-4110-a069-38c79809bc98","Type":"ContainerStarted","Data":"fa0f023ed372bf167ac295cb3f6bb66405aa557ac96199396087a4d95ecb9d6c"} Oct 30 21:09:03.947249 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:03.946858 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" event={"ID":"9cbcfa4c-ad1b-4110-a069-38c79809bc98","Type":"ContainerStarted","Data":"39671291116a48a1b204098de72ac7a92ac7bf74c92a12d8d1f772eef06b3794"} Oct 30 21:09:03.994057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:03.993856 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" podStartSLOduration=1.993839215 podStartE2EDuration="1.993839215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:09:03.9933906 +0000 UTC m=+531.114464347" watchObservedRunningTime="2026-10-30 21:09:03.993839215 +0000 UTC m=+531.114912966" Oct 30 21:09:04.729534 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:04.729499 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:04.730870 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:04.730848 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cdm7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 21:09:04.730927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:04.730907 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" podUID="9cbcfa4c-ad1b-4110-a069-38c79809bc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 21:09:05.729652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:05.729573 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cdm7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 21:09:05.729652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:05.729628 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" podUID="9cbcfa4c-ad1b-4110-a069-38c79809bc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 21:09:06.729612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:06.729573 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cdm7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 21:09:06.729802 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:06.729629 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" podUID="9cbcfa4c-ad1b-4110-a069-38c79809bc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 21:09:07.502405 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:09:07.502370 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:09:07.729546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:07.729507 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cdm7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 21:09:07.729717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:07.729565 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" podUID="9cbcfa4c-ad1b-4110-a069-38c79809bc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 21:09:08.729419 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:08.729383 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cdm7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 21:09:08.729899 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:08.729434 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" podUID="9cbcfa4c-ad1b-4110-a069-38c79809bc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 21:09:09.729426 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:09.729390 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cdm7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Oct 30 21:09:09.729818 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:09.729450 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" podUID="9cbcfa4c-ad1b-4110-a069-38c79809bc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Oct 30 21:09:11.260849 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:11.260811 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:11.261325 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:11.261215 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:11.262177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:11.262155 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:09:37.483446 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:09:37.483406 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:09:56.701449 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:56.701371 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:09:57.308008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:57.307974 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm"] Oct 30 21:09:57.311318 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:57.311303 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm" Oct 30 21:09:57.330829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:57.330797 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm"] Oct 30 21:09:57.338319 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:57.338295 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-2cda44\"" Oct 30 21:09:57.459282 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:57.459249 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx6g2\" (UniqueName: \"kubernetes.io/projected/73ef630c-e7d0-4b76-b194-792276f55a67-kube-api-access-mx6g2\") pod \"payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm\" (UID: \"73ef630c-e7d0-4b76-b194-792276f55a67\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm" Oct 30 21:09:57.459439 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:57.459396 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73ef630c-e7d0-4b76-b194-792276f55a67-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm\" (UID: \"73ef630c-e7d0-4b76-b194-792276f55a67\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm" Oct 30 21:09:57.560900 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:57.560797 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73ef630c-e7d0-4b76-b194-792276f55a67-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm\" (UID: \"73ef630c-e7d0-4b76-b194-792276f55a67\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm" Oct 30 21:09:57.560900 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:57.560862 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx6g2\" (UniqueName: \"kubernetes.io/projected/73ef630c-e7d0-4b76-b194-792276f55a67-kube-api-access-mx6g2\") pod \"payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm\" (UID: \"73ef630c-e7d0-4b76-b194-792276f55a67\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm" Oct 30 21:09:57.561655 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:57.561629 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73ef630c-e7d0-4b76-b194-792276f55a67-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm\" (UID: \"73ef630c-e7d0-4b76-b194-792276f55a67\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm" Oct 30 21:09:57.579228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:57.579196 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx6g2\" (UniqueName: \"kubernetes.io/projected/73ef630c-e7d0-4b76-b194-792276f55a67-kube-api-access-mx6g2\") pod \"payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm\" (UID: \"73ef630c-e7d0-4b76-b194-792276f55a67\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm" Oct 30 21:09:57.625365 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:57.625313 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm" Oct 30 21:09:57.930000 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:57.929956 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm"] Oct 30 21:09:57.934007 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:09:57.933977 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73ef630c_e7d0_4b76_b194_792276f55a67.slice/crio-2d6b41827405e17d49d22051f3d961b27c517cb6a8238c53f881bb2bba527bc1 WatchSource:0}: Error finding container 2d6b41827405e17d49d22051f3d961b27c517cb6a8238c53f881bb2bba527bc1: Status 404 returned error can't find the container with id 2d6b41827405e17d49d22051f3d961b27c517cb6a8238c53f881bb2bba527bc1 Oct 30 21:09:58.189063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.188973 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92"] Oct 30 21:09:58.193788 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.193763 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92" Oct 30 21:09:58.208804 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.208779 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-839163-dockercfg-tmgjb\"" Oct 30 21:09:58.208950 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.208809 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-839163\"" Oct 30 21:09:58.212210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.212168 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92"] Oct 30 21:09:58.230710 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.230683 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft"] Oct 30 21:09:58.234685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.234662 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft" Oct 30 21:09:58.241598 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.241557 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm" event={"ID":"73ef630c-e7d0-4b76-b194-792276f55a67","Type":"ContainerStarted","Data":"2d6b41827405e17d49d22051f3d961b27c517cb6a8238c53f881bb2bba527bc1"} Oct 30 21:09:58.252925 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.252896 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft"] Oct 30 21:09:58.369780 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.369749 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11be824c-778f-4301-b2c5-7b0c8ca3d04c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft\" (UID: \"11be824c-778f-4301-b2c5-7b0c8ca3d04c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft" Oct 30 21:09:58.369977 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.369792 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rwwf\" (UniqueName: \"kubernetes.io/projected/11be824c-778f-4301-b2c5-7b0c8ca3d04c-kube-api-access-2rwwf\") pod \"payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft\" (UID: \"11be824c-778f-4301-b2c5-7b0c8ca3d04c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft" Oct 30 21:09:58.369977 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.369953 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/586f88e9-d006-46ea-bef3-68cc117aaa81-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92\" (UID: \"586f88e9-d006-46ea-bef3-68cc117aaa81\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92" Oct 30 21:09:58.370122 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.370006 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwtpc\" (UniqueName: \"kubernetes.io/projected/586f88e9-d006-46ea-bef3-68cc117aaa81-kube-api-access-cwtpc\") pod \"payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92\" (UID: \"586f88e9-d006-46ea-bef3-68cc117aaa81\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92" Oct 30 21:09:58.470887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.470811 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rwwf\" (UniqueName: \"kubernetes.io/projected/11be824c-778f-4301-b2c5-7b0c8ca3d04c-kube-api-access-2rwwf\") pod \"payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft\" (UID: \"11be824c-778f-4301-b2c5-7b0c8ca3d04c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft" Oct 30 21:09:58.470887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.470880 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/586f88e9-d006-46ea-bef3-68cc117aaa81-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92\" (UID: \"586f88e9-d006-46ea-bef3-68cc117aaa81\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92" Oct 30 21:09:58.471156 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.470906 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwtpc\" (UniqueName: \"kubernetes.io/projected/586f88e9-d006-46ea-bef3-68cc117aaa81-kube-api-access-cwtpc\") pod \"payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92\" (UID: \"586f88e9-d006-46ea-bef3-68cc117aaa81\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92" Oct 30 21:09:58.471156 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.470936 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11be824c-778f-4301-b2c5-7b0c8ca3d04c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft\" (UID: \"11be824c-778f-4301-b2c5-7b0c8ca3d04c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft" Oct 30 21:09:58.471654 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.471634 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11be824c-778f-4301-b2c5-7b0c8ca3d04c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft\" (UID: \"11be824c-778f-4301-b2c5-7b0c8ca3d04c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft" Oct 30 21:09:58.471692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.471670 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/586f88e9-d006-46ea-bef3-68cc117aaa81-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92\" (UID: \"586f88e9-d006-46ea-bef3-68cc117aaa81\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92" Oct 30 21:09:58.488658 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.488633 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rwwf\" (UniqueName: \"kubernetes.io/projected/11be824c-778f-4301-b2c5-7b0c8ca3d04c-kube-api-access-2rwwf\") pod \"payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft\" (UID: \"11be824c-778f-4301-b2c5-7b0c8ca3d04c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft" Oct 30 21:09:58.488752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.488697 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwtpc\" (UniqueName: \"kubernetes.io/projected/586f88e9-d006-46ea-bef3-68cc117aaa81-kube-api-access-cwtpc\") pod \"payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92\" (UID: \"586f88e9-d006-46ea-bef3-68cc117aaa81\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92" Oct 30 21:09:58.507166 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.507143 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92" Oct 30 21:09:58.555662 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.555369 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft" Oct 30 21:09:58.813311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.813234 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92"] Oct 30 21:09:58.816580 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:09:58.816543 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586f88e9_d006_46ea_bef3_68cc117aaa81.slice/crio-8ecc39eec06a3e28f33daeaf601a6cb554d66f8d1a56f53f3c9a5938f771bf24 WatchSource:0}: Error finding container 8ecc39eec06a3e28f33daeaf601a6cb554d66f8d1a56f53f3c9a5938f771bf24: Status 404 returned error can't find the container with id 8ecc39eec06a3e28f33daeaf601a6cb554d66f8d1a56f53f3c9a5938f771bf24 Oct 30 21:09:58.884632 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:58.884595 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft"] Oct 30 21:09:58.889433 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:09:58.889397 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11be824c_778f_4301_b2c5_7b0c8ca3d04c.slice/crio-dbc6c5ac60af8d82e0e7caedbe8610124ec7a1bef7420643df3fc4ccaf863967 WatchSource:0}: Error finding container dbc6c5ac60af8d82e0e7caedbe8610124ec7a1bef7420643df3fc4ccaf863967: Status 404 returned error can't find the container with id dbc6c5ac60af8d82e0e7caedbe8610124ec7a1bef7420643df3fc4ccaf863967 Oct 30 21:09:59.251239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:59.251196 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft" event={"ID":"11be824c-778f-4301-b2c5-7b0c8ca3d04c","Type":"ContainerStarted","Data":"dbc6c5ac60af8d82e0e7caedbe8610124ec7a1bef7420643df3fc4ccaf863967"} Oct 30 21:09:59.254434 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:09:59.254401 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92" event={"ID":"586f88e9-d006-46ea-bef3-68cc117aaa81","Type":"ContainerStarted","Data":"8ecc39eec06a3e28f33daeaf601a6cb554d66f8d1a56f53f3c9a5938f771bf24"} Oct 30 21:10:01.269393 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:01.269355 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm" event={"ID":"73ef630c-e7d0-4b76-b194-792276f55a67","Type":"ContainerStarted","Data":"571d580e9ca5ccb32eecb779c674cf7269dbd93e9151569fd5643f054e703a02"} Oct 30 21:10:01.269764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:01.269420 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm" Oct 30 21:10:02.279649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:02.279603 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92" event={"ID":"586f88e9-d006-46ea-bef3-68cc117aaa81","Type":"ContainerStarted","Data":"1428cc18e61ee2b52e2b0c2de322a8bcd85295aa31c4dda54b09bbc5de0295b1"} Oct 30 21:10:02.280240 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:02.279874 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92" Oct 30 21:10:02.281877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:02.281853 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft" event={"ID":"11be824c-778f-4301-b2c5-7b0c8ca3d04c","Type":"ContainerStarted","Data":"42e68838bfebe24dff828eb4163541fa2404849a78b58a906e0f21ac908e3df0"} Oct 30 21:10:02.282066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:02.282048 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft" Oct 30 21:10:02.317813 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:02.317770 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm" podStartSLOduration=2.544218091 podStartE2EDuration="5.317757731s" totalImagesPullingTime="2.77353964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:01.309361052 +0000 UTC m=+588.430434799" watchObservedRunningTime="2026-10-30 21:10:02.317757731 +0000 UTC m=+589.438831476" Oct 30 21:10:02.318508 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:02.318479 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92" podStartSLOduration=1.8635059950000001 podStartE2EDuration="4.31847107s" totalImagesPullingTime="2.454965075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:02.316770466 +0000 UTC m=+589.437844214" watchObservedRunningTime="2026-10-30 21:10:02.31847107 +0000 UTC m=+589.439544817" Oct 30 21:10:02.354184 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:02.354146 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft" podStartSLOduration=1.948070357 podStartE2EDuration="4.35413458s" totalImagesPullingTime="2.406064223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:02.352867647 +0000 UTC m=+589.473941393" watchObservedRunningTime="2026-10-30 21:10:02.35413458 +0000 UTC m=+589.475208325" Oct 30 21:10:02.550071 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:02.549957 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:10:04.403186 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:04.403149 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl"] Oct 30 21:10:04.406702 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:04.406685 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" Oct 30 21:10:04.423284 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:04.423257 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4pdl4\"" Oct 30 21:10:04.426066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:04.426027 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl"] Oct 30 21:10:04.434339 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:04.434314 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:10:04.434473 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:04.434348 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:10:04.456146 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:04.456118 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mccd\" (UniqueName: \"kubernetes.io/projected/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-kube-api-access-4mccd\") pod \"maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl\" (UID: \"e0dfe0cb-0115-431c-acd8-ac154a18a0ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" Oct 30 21:10:04.456331 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:04.456192 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl\" (UID: \"e0dfe0cb-0115-431c-acd8-ac154a18a0ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" Oct 30 21:10:04.456331 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:04.456255 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl\" (UID: \"e0dfe0cb-0115-431c-acd8-ac154a18a0ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" Oct 30 21:10:04.456435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:04.456352 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-metrics-certs\") pod \"maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl\" (UID: \"e0dfe0cb-0115-431c-acd8-ac154a18a0ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" Oct 30 21:10:04.518369 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:04.518344 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-2cda44-serving-cert\"" Oct 30 21:10:04.518369 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:04.518359 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:10:04.557830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:04.557793 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl\" (UID: \"e0dfe0cb-0115-431c-acd8-ac154a18a0ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" Oct 30 21:10:04.557974 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:04.557866 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl\" (UID: \"e0dfe0cb-0115-431c-acd8-ac154a18a0ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" Oct 30 21:10:04.557974 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:04.557884 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-metrics-certs\") pod \"maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl\" (UID: \"e0dfe0cb-0115-431c-acd8-ac154a18a0ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" Oct 30 21:10:04.557974 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:04.557939 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mccd\" (UniqueName: \"kubernetes.io/projected/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-kube-api-access-4mccd\") pod \"maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl\" (UID: \"e0dfe0cb-0115-431c-acd8-ac154a18a0ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" Oct 30 21:10:04.558551 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:04.558525 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl\" (UID: \"e0dfe0cb-0115-431c-acd8-ac154a18a0ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" Oct 30 21:10:04.560371 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:04.560354 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-maas-api-tls\") pod \"maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl\" (UID: \"e0dfe0cb-0115-431c-acd8-ac154a18a0ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" Oct 30 21:10:04.560461 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:04.560420 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-metrics-certs\") pod \"maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl\" (UID: \"e0dfe0cb-0115-431c-acd8-ac154a18a0ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" Oct 30 21:10:04.577908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:04.577884 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mccd\" (UniqueName: \"kubernetes.io/projected/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-kube-api-access-4mccd\") pod \"maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl\" (UID: \"e0dfe0cb-0115-431c-acd8-ac154a18a0ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" Oct 30 21:10:04.720760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:04.720727 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" Oct 30 21:10:05.017383 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:05.017358 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl"] Oct 30 21:10:05.019886 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:10:05.019860 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0dfe0cb_0115_431c_acd8_ac154a18a0ee.slice/crio-55c838f5a74dc9a6d865c38a65d41bd4a17c8c635e87760b5091de2cbaeac038 WatchSource:0}: Error finding container 55c838f5a74dc9a6d865c38a65d41bd4a17c8c635e87760b5091de2cbaeac038: Status 404 returned error can't find the container with id 55c838f5a74dc9a6d865c38a65d41bd4a17c8c635e87760b5091de2cbaeac038 Oct 30 21:10:05.300369 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:05.300290 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" event={"ID":"e0dfe0cb-0115-431c-acd8-ac154a18a0ee","Type":"ContainerStarted","Data":"55c838f5a74dc9a6d865c38a65d41bd4a17c8c635e87760b5091de2cbaeac038"} Oct 30 21:10:07.484390 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:10:07.484355 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:10:08.320225 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:08.320184 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" event={"ID":"e0dfe0cb-0115-431c-acd8-ac154a18a0ee","Type":"ContainerStarted","Data":"cbd9ee9d09878ee206bb1a7f2f55fea1f65f8a098f0bca966d5460cf3ffa7255"} Oct 30 21:10:08.320420 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:08.320351 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" Oct 30 21:10:08.360425 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:08.360371 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" podStartSLOduration=2.062355361 podStartE2EDuration="4.360355844s" totalImagesPullingTime="2.298000483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:08.358899631 +0000 UTC m=+595.479973368" watchObservedRunningTime="2026-10-30 21:10:08.360355844 +0000 UTC m=+595.481429590" Oct 30 21:10:09.373211 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:09.373170 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:10:10.460548 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.460516 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws"] Oct 30 21:10:10.465175 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.465152 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:10.482542 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.482506 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws"] Oct 30 21:10:10.489801 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.489764 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-56ac74\"/\"openshift-service-ca.crt\"" Oct 30 21:10:10.489946 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.489841 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-56ac74\"/\"default-dockercfg-rhfqr\"" Oct 30 21:10:10.490109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.490086 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-56ac74\"/\"kube-root-ca.crt\"" Oct 30 21:10:10.491171 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.491152 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-56ac74\"/\"simulator-w1-56ac74-kserve-self-signed-certs\"" Oct 30 21:10:10.514709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.514680 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-home\") pod \"simulator-w1-56ac74-kserve-df666c46d-pw8ws\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:10.514852 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.514737 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-dshm\") pod \"simulator-w1-56ac74-kserve-df666c46d-pw8ws\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:10.514852 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.514815 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbhfh\" (UniqueName: \"kubernetes.io/projected/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-kube-api-access-sbhfh\") pod \"simulator-w1-56ac74-kserve-df666c46d-pw8ws\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:10.514944 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.514853 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-model-cache\") pod \"simulator-w1-56ac74-kserve-df666c46d-pw8ws\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:10.514944 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.514930 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-tmp-dir\") pod \"simulator-w1-56ac74-kserve-df666c46d-pw8ws\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:10.515061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.514990 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-tls-certs\") pod \"simulator-w1-56ac74-kserve-df666c46d-pw8ws\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:10.616275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.616233 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-home\") pod \"simulator-w1-56ac74-kserve-df666c46d-pw8ws\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:10.616474 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.616300 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-dshm\") pod \"simulator-w1-56ac74-kserve-df666c46d-pw8ws\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:10.616549 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.616467 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbhfh\" (UniqueName: \"kubernetes.io/projected/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-kube-api-access-sbhfh\") pod \"simulator-w1-56ac74-kserve-df666c46d-pw8ws\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:10.616798 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.616772 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-home\") pod \"simulator-w1-56ac74-kserve-df666c46d-pw8ws\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:10.616906 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.616846 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-model-cache\") pod \"simulator-w1-56ac74-kserve-df666c46d-pw8ws\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:10.616906 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.616895 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-tmp-dir\") pod \"simulator-w1-56ac74-kserve-df666c46d-pw8ws\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:10.617053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.616966 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-tls-certs\") pod \"simulator-w1-56ac74-kserve-df666c46d-pw8ws\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:10.617249 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.617225 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-tmp-dir\") pod \"simulator-w1-56ac74-kserve-df666c46d-pw8ws\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:10.617469 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.617437 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-model-cache\") pod \"simulator-w1-56ac74-kserve-df666c46d-pw8ws\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:10.619107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.619084 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-dshm\") pod \"simulator-w1-56ac74-kserve-df666c46d-pw8ws\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:10.619768 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.619745 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-tls-certs\") pod \"simulator-w1-56ac74-kserve-df666c46d-pw8ws\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:10.634670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.634635 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbhfh\" (UniqueName: \"kubernetes.io/projected/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-kube-api-access-sbhfh\") pod \"simulator-w1-56ac74-kserve-df666c46d-pw8ws\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:10.780871 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:10.780792 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:11.080211 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:11.080183 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws"] Oct 30 21:10:11.081977 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:10:11.081947 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1f7f668_8bb1_435b_a2d8_7340fc3b7c32.slice/crio-e25e0d45311a0842c82b962ffd57a5e9efde7616cdaa125200239d44f13e16c7 WatchSource:0}: Error finding container e25e0d45311a0842c82b962ffd57a5e9efde7616cdaa125200239d44f13e16c7: Status 404 returned error can't find the container with id e25e0d45311a0842c82b962ffd57a5e9efde7616cdaa125200239d44f13e16c7 Oct 30 21:10:11.340650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:11.340561 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" event={"ID":"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32","Type":"ContainerStarted","Data":"e25e0d45311a0842c82b962ffd57a5e9efde7616cdaa125200239d44f13e16c7"} Oct 30 21:10:12.285650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:12.285623 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm" Oct 30 21:10:12.353270 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:12.353222 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" event={"ID":"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32","Type":"ContainerStarted","Data":"a8e8991031872ef634a232d625e7470c8539b3ad1f885726de699b922926aaf9"} Oct 30 21:10:12.395863 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:12.395804 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podStartSLOduration=1.277043417 podStartE2EDuration="2.395785538s" totalImagesPullingTime="1.118742121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:12.393099343 +0000 UTC m=+599.514173086" watchObservedRunningTime="2026-10-30 21:10:12.395785538 +0000 UTC m=+599.516859287" Oct 30 21:10:12.781922 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:12.781889 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:13.100089 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:13.099998 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:10:13.100498 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:13.100465 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:10:13.290998 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:13.290969 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft" Oct 30 21:10:13.291462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:13.291154 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92" Oct 30 21:10:14.332641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:14.332608 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" Oct 30 21:10:14.763184 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:14.763153 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:10:20.781547 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:20.781514 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:20.794560 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:20.794532 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:20.934846 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:20.934811 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh"] Oct 30 21:10:20.940630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:20.940605 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:20.955850 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:20.955816 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-79477e\"/\"kube-root-ca.crt\"" Oct 30 21:10:20.956002 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:20.955965 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-79477e\"/\"default-dockercfg-74c78\"" Oct 30 21:10:20.956147 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:20.955993 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-79477e\"/\"openshift-service-ca.crt\"" Oct 30 21:10:20.956278 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:20.956258 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-79477e\"/\"simulator-w3-79477e-kserve-self-signed-certs\"" Oct 30 21:10:20.957895 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:20.957873 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh"] Oct 30 21:10:21.021812 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:21.021782 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-home\") pod \"simulator-w3-79477e-kserve-6b747588b5-hfxrh\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:21.021963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:21.021818 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e190db51-e133-4ca2-a26b-f89c2b67da45-tls-certs\") pod \"simulator-w3-79477e-kserve-6b747588b5-hfxrh\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:21.021963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:21.021895 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c98r\" (UniqueName: \"kubernetes.io/projected/e190db51-e133-4ca2-a26b-f89c2b67da45-kube-api-access-8c98r\") pod \"simulator-w3-79477e-kserve-6b747588b5-hfxrh\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:21.021963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:21.021921 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-model-cache\") pod \"simulator-w3-79477e-kserve-6b747588b5-hfxrh\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:21.022081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:21.022054 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-dshm\") pod \"simulator-w3-79477e-kserve-6b747588b5-hfxrh\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:21.022116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:21.022080 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-tmp-dir\") pod \"simulator-w3-79477e-kserve-6b747588b5-hfxrh\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:21.123260 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:21.123179 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c98r\" (UniqueName: \"kubernetes.io/projected/e190db51-e133-4ca2-a26b-f89c2b67da45-kube-api-access-8c98r\") pod \"simulator-w3-79477e-kserve-6b747588b5-hfxrh\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:21.123260 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:21.123224 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-model-cache\") pod \"simulator-w3-79477e-kserve-6b747588b5-hfxrh\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:21.123488 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:21.123281 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-dshm\") pod \"simulator-w3-79477e-kserve-6b747588b5-hfxrh\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:21.123488 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:21.123296 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-tmp-dir\") pod \"simulator-w3-79477e-kserve-6b747588b5-hfxrh\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:21.123592 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:21.123567 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-home\") pod \"simulator-w3-79477e-kserve-6b747588b5-hfxrh\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:21.123651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:21.123606 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e190db51-e133-4ca2-a26b-f89c2b67da45-tls-certs\") pod \"simulator-w3-79477e-kserve-6b747588b5-hfxrh\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:21.123780 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:21.123753 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-model-cache\") pod \"simulator-w3-79477e-kserve-6b747588b5-hfxrh\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:21.123894 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:21.123879 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-tmp-dir\") pod \"simulator-w3-79477e-kserve-6b747588b5-hfxrh\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:21.123957 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:21.123912 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-home\") pod \"simulator-w3-79477e-kserve-6b747588b5-hfxrh\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:21.125810 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:21.125782 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-dshm\") pod \"simulator-w3-79477e-kserve-6b747588b5-hfxrh\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:21.126056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:21.126017 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e190db51-e133-4ca2-a26b-f89c2b67da45-tls-certs\") pod \"simulator-w3-79477e-kserve-6b747588b5-hfxrh\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:21.141945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:21.141914 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c98r\" (UniqueName: \"kubernetes.io/projected/e190db51-e133-4ca2-a26b-f89c2b67da45-kube-api-access-8c98r\") pod \"simulator-w3-79477e-kserve-6b747588b5-hfxrh\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:21.255146 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:21.255111 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:21.417074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:21.417046 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:10:21.560740 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:21.560717 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh"] Oct 30 21:10:21.562520 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:10:21.562491 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode190db51_e133_4ca2_a26b_f89c2b67da45.slice/crio-f1ba8e2df61dba21fde115c624f6e3d43ad41d2e933132d3993ec9b4e723b330 WatchSource:0}: Error finding container f1ba8e2df61dba21fde115c624f6e3d43ad41d2e933132d3993ec9b4e723b330: Status 404 returned error can't find the container with id f1ba8e2df61dba21fde115c624f6e3d43ad41d2e933132d3993ec9b4e723b330 Oct 30 21:10:22.413902 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:22.413860 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" event={"ID":"e190db51-e133-4ca2-a26b-f89c2b67da45","Type":"ContainerStarted","Data":"8c065bf7415ba4bd99b83fde94989ae6a27de60325cc340d5d781f6ba5aa8a59"} Oct 30 21:10:22.414381 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:22.413914 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" event={"ID":"e190db51-e133-4ca2-a26b-f89c2b67da45","Type":"ContainerStarted","Data":"f1ba8e2df61dba21fde115c624f6e3d43ad41d2e933132d3993ec9b4e723b330"} Oct 30 21:10:22.453361 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:22.453306 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podStartSLOduration=2.4532885540000002 podStartE2EDuration="2.453288554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:22.451095688 +0000 UTC m=+609.572169438" watchObservedRunningTime="2026-10-30 21:10:22.453288554 +0000 UTC m=+609.574362300" Oct 30 21:10:23.255328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:23.255298 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:25.217001 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.216969 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm"] Oct 30 21:10:25.220915 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.220889 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:25.249929 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.249899 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-839163\"/\"openshift-service-ca.crt\"" Oct 30 21:10:25.250090 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.250071 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-839163\"/\"kube-root-ca.crt\"" Oct 30 21:10:25.252194 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.252174 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm"] Oct 30 21:10:25.284357 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.284330 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-839163\"/\"simulator-w4-839163-kserve-self-signed-certs\"" Oct 30 21:10:25.284528 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.284399 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-839163\"/\"default-dockercfg-xgml5\"" Oct 30 21:10:25.362281 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.362245 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8twjb\" (UniqueName: \"kubernetes.io/projected/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-kube-api-access-8twjb\") pod \"simulator-w4-839163-kserve-5c66cb4d9f-sf5fm\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:25.362467 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.362316 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-model-cache\") pod \"simulator-w4-839163-kserve-5c66cb4d9f-sf5fm\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:25.362542 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.362466 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-dshm\") pod \"simulator-w4-839163-kserve-5c66cb4d9f-sf5fm\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:25.362602 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.362565 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-tmp-dir\") pod \"simulator-w4-839163-kserve-5c66cb4d9f-sf5fm\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:25.362684 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.362657 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-tls-certs\") pod \"simulator-w4-839163-kserve-5c66cb4d9f-sf5fm\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:25.362827 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.362695 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-home\") pod \"simulator-w4-839163-kserve-5c66cb4d9f-sf5fm\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:25.463709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.463673 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-model-cache\") pod \"simulator-w4-839163-kserve-5c66cb4d9f-sf5fm\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:25.463709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.463710 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-dshm\") pod \"simulator-w4-839163-kserve-5c66cb4d9f-sf5fm\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:25.463933 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.463752 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-tmp-dir\") pod \"simulator-w4-839163-kserve-5c66cb4d9f-sf5fm\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:25.463933 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.463787 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-tls-certs\") pod \"simulator-w4-839163-kserve-5c66cb4d9f-sf5fm\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:25.463933 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.463811 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-home\") pod \"simulator-w4-839163-kserve-5c66cb4d9f-sf5fm\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:25.463933 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.463875 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8twjb\" (UniqueName: \"kubernetes.io/projected/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-kube-api-access-8twjb\") pod \"simulator-w4-839163-kserve-5c66cb4d9f-sf5fm\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:25.464243 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.464215 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-model-cache\") pod \"simulator-w4-839163-kserve-5c66cb4d9f-sf5fm\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:25.464371 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.464229 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-tmp-dir\") pod \"simulator-w4-839163-kserve-5c66cb4d9f-sf5fm\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:25.464446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.464403 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-home\") pod \"simulator-w4-839163-kserve-5c66cb4d9f-sf5fm\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:25.466155 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.466133 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-dshm\") pod \"simulator-w4-839163-kserve-5c66cb4d9f-sf5fm\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:25.466555 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.466533 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-tls-certs\") pod \"simulator-w4-839163-kserve-5c66cb4d9f-sf5fm\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:25.485742 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.485683 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8twjb\" (UniqueName: \"kubernetes.io/projected/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-kube-api-access-8twjb\") pod \"simulator-w4-839163-kserve-5c66cb4d9f-sf5fm\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:25.536191 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.536159 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:25.854668 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.854640 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm"] Oct 30 21:10:25.856687 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:10:25.856660 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbbb5e6e6_a437_491d_9d1c_a1bfe6b258f4.slice/crio-6c6d16153b84bfb98e35d37a8f08402c6893df248469d2a07b20a583d296f9fc WatchSource:0}: Error finding container 6c6d16153b84bfb98e35d37a8f08402c6893df248469d2a07b20a583d296f9fc: Status 404 returned error can't find the container with id 6c6d16153b84bfb98e35d37a8f08402c6893df248469d2a07b20a583d296f9fc Oct 30 21:10:25.944155 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:25.944118 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:10:26.441404 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:26.441365 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" event={"ID":"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4","Type":"ContainerStarted","Data":"b7fb8a889ab78ee1dbe8968ffd176fe7a6866e51d2b1bd1121af6c180209e9a2"} Oct 30 21:10:26.441404 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:26.441410 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" event={"ID":"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4","Type":"ContainerStarted","Data":"6c6d16153b84bfb98e35d37a8f08402c6893df248469d2a07b20a583d296f9fc"} Oct 30 21:10:26.495495 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:26.495442 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podStartSLOduration=1.495428286 podStartE2EDuration="1.495428286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:26.495364034 +0000 UTC m=+613.616437780" watchObservedRunningTime="2026-10-30 21:10:26.495428286 +0000 UTC m=+613.616502034" Oct 30 21:10:26.537108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:26.537082 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:31.255721 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:31.255678 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:31.272488 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:31.272463 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:31.488837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:31.488812 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:10:34.945794 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:34.945762 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:10:35.221624 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.221546 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx"] Oct 30 21:10:35.230208 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.230177 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:35.243441 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.243412 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx"] Oct 30 21:10:35.250976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.250944 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-43c070\"/\"openshift-service-ca.crt\"" Oct 30 21:10:35.254643 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.254620 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-43c070\"/\"default-dockercfg-w4c7h\"" Oct 30 21:10:35.254643 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.254634 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-43c070\"/\"kube-root-ca.crt\"" Oct 30 21:10:35.254876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.254864 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-43c070\"/\"simulator-w5-43c070-kserve-self-signed-certs\"" Oct 30 21:10:35.361623 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.361590 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8733709-97b3-4b3a-85cd-f47c62a9fc71-tls-certs\") pod \"simulator-w5-43c070-kserve-f858d6fd7-z27jx\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:35.361798 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.361638 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-dshm\") pod \"simulator-w5-43c070-kserve-f858d6fd7-z27jx\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:35.361798 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.361752 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq9nk\" (UniqueName: \"kubernetes.io/projected/e8733709-97b3-4b3a-85cd-f47c62a9fc71-kube-api-access-qq9nk\") pod \"simulator-w5-43c070-kserve-f858d6fd7-z27jx\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:35.361798 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.361786 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-model-cache\") pod \"simulator-w5-43c070-kserve-f858d6fd7-z27jx\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:35.361936 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.361852 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-tmp-dir\") pod \"simulator-w5-43c070-kserve-f858d6fd7-z27jx\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:35.361936 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.361893 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-home\") pod \"simulator-w5-43c070-kserve-f858d6fd7-z27jx\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:35.463506 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.463461 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8733709-97b3-4b3a-85cd-f47c62a9fc71-tls-certs\") pod \"simulator-w5-43c070-kserve-f858d6fd7-z27jx\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:35.463712 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.463521 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-dshm\") pod \"simulator-w5-43c070-kserve-f858d6fd7-z27jx\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:35.463712 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.463582 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq9nk\" (UniqueName: \"kubernetes.io/projected/e8733709-97b3-4b3a-85cd-f47c62a9fc71-kube-api-access-qq9nk\") pod \"simulator-w5-43c070-kserve-f858d6fd7-z27jx\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:35.463712 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.463612 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-model-cache\") pod \"simulator-w5-43c070-kserve-f858d6fd7-z27jx\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:35.463712 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.463660 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-tmp-dir\") pod \"simulator-w5-43c070-kserve-f858d6fd7-z27jx\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:35.463712 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.463705 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-home\") pod \"simulator-w5-43c070-kserve-f858d6fd7-z27jx\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:35.464203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.464180 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-home\") pod \"simulator-w5-43c070-kserve-f858d6fd7-z27jx\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:35.466427 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.466403 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-model-cache\") pod \"simulator-w5-43c070-kserve-f858d6fd7-z27jx\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:35.466981 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.466880 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-tmp-dir\") pod \"simulator-w5-43c070-kserve-f858d6fd7-z27jx\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:35.467416 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.467390 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8733709-97b3-4b3a-85cd-f47c62a9fc71-tls-certs\") pod \"simulator-w5-43c070-kserve-f858d6fd7-z27jx\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:35.471269 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.471245 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-dshm\") pod \"simulator-w5-43c070-kserve-f858d6fd7-z27jx\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:35.487714 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.487647 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq9nk\" (UniqueName: \"kubernetes.io/projected/e8733709-97b3-4b3a-85cd-f47c62a9fc71-kube-api-access-qq9nk\") pod \"simulator-w5-43c070-kserve-f858d6fd7-z27jx\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:35.536862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.536826 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:35.547451 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.547428 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:35.549773 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.549750 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:35.847694 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:35.847561 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx"] Oct 30 21:10:35.849976 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:10:35.849943 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode8733709_97b3_4b3a_85cd_f47c62a9fc71.slice/crio-6886b0055a88088c409394f2727236b7d4da4259fcb0811c5fd7c03ad4a17fff WatchSource:0}: Error finding container 6886b0055a88088c409394f2727236b7d4da4259fcb0811c5fd7c03ad4a17fff: Status 404 returned error can't find the container with id 6886b0055a88088c409394f2727236b7d4da4259fcb0811c5fd7c03ad4a17fff Oct 30 21:10:36.504552 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:36.504519 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" event={"ID":"e8733709-97b3-4b3a-85cd-f47c62a9fc71","Type":"ContainerStarted","Data":"3f52ba3f3a4e0075151a30ea90011884e91592b976da0d103bdbdc0db2dcc992"} Oct 30 21:10:36.505011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:36.504569 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" event={"ID":"e8733709-97b3-4b3a-85cd-f47c62a9fc71","Type":"ContainerStarted","Data":"6886b0055a88088c409394f2727236b7d4da4259fcb0811c5fd7c03ad4a17fff"} Oct 30 21:10:36.515842 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:36.515813 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:10:36.544554 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:36.544502 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podStartSLOduration=1.5444884060000001 podStartE2EDuration="1.544488406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:36.544336645 +0000 UTC m=+623.665410391" watchObservedRunningTime="2026-10-30 21:10:36.544488406 +0000 UTC m=+623.665562152" Oct 30 21:10:36.549125 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:36.549072 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:37.486699 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:10:37.486660 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:10:41.253562 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.253525 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc"] Oct 30 21:10:41.261438 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.261402 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:41.276941 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.276913 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc"] Oct 30 21:10:41.285560 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.285535 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ed088d\"/\"default-dockercfg-p7744\"" Oct 30 21:10:41.285687 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.285654 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ed088d\"/\"simulator-w6-ed088d-kserve-self-signed-certs\"" Oct 30 21:10:41.286468 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.286445 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ed088d\"/\"openshift-service-ca.crt\"" Oct 30 21:10:41.286581 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.286526 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ed088d\"/\"kube-root-ca.crt\"" Oct 30 21:10:41.425592 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.425557 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-model-cache\") pod \"simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:41.425592 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.425596 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-dshm\") pod \"simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:41.425804 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.425669 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j59bv\" (UniqueName: \"kubernetes.io/projected/1cc79717-77ce-4405-b102-ff904f2b345f-kube-api-access-j59bv\") pod \"simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:41.425804 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.425765 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-tmp-dir\") pod \"simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:41.425900 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.425880 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-home\") pod \"simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:41.425966 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.425950 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc79717-77ce-4405-b102-ff904f2b345f-tls-certs\") pod \"simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:41.526853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.526762 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j59bv\" (UniqueName: \"kubernetes.io/projected/1cc79717-77ce-4405-b102-ff904f2b345f-kube-api-access-j59bv\") pod \"simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:41.526853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.526817 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-tmp-dir\") pod \"simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:41.527126 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.526924 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-home\") pod \"simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:41.527126 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.526987 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc79717-77ce-4405-b102-ff904f2b345f-tls-certs\") pod \"simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:41.527126 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.527077 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-model-cache\") pod \"simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:41.527126 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.527109 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-dshm\") pod \"simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:41.527342 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.527301 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-tmp-dir\") pod \"simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:41.527628 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.527603 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-model-cache\") pod \"simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:41.527782 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.527765 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-home\") pod \"simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:41.529950 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.529923 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc79717-77ce-4405-b102-ff904f2b345f-tls-certs\") pod \"simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:41.531106 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.531085 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-dshm\") pod \"simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:41.554137 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.554108 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j59bv\" (UniqueName: \"kubernetes.io/projected/1cc79717-77ce-4405-b102-ff904f2b345f-kube-api-access-j59bv\") pod \"simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:41.578785 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.578747 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:41.876066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:41.876016 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc"] Oct 30 21:10:41.879344 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:10:41.879315 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1cc79717_77ce_4405_b102_ff904f2b345f.slice/crio-f64ece6ba5c991549d413be4908e19bc226654b01b0c7fc4f58f24841ec93513 WatchSource:0}: Error finding container f64ece6ba5c991549d413be4908e19bc226654b01b0c7fc4f58f24841ec93513: Status 404 returned error can't find the container with id f64ece6ba5c991549d413be4908e19bc226654b01b0c7fc4f58f24841ec93513 Oct 30 21:10:42.546674 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:42.546635 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" event={"ID":"1cc79717-77ce-4405-b102-ff904f2b345f","Type":"ContainerStarted","Data":"4870892cdb213c59c1442b76b7c1c3d45a1f2921f7c88f5c8205826ce573240e"} Oct 30 21:10:42.547170 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:42.546684 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" event={"ID":"1cc79717-77ce-4405-b102-ff904f2b345f","Type":"ContainerStarted","Data":"f64ece6ba5c991549d413be4908e19bc226654b01b0c7fc4f58f24841ec93513"} Oct 30 21:10:42.579829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:42.579785 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:42.583648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:42.583602 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podStartSLOduration=1.583589444 podStartE2EDuration="1.583589444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:42.581870785 +0000 UTC m=+629.702944513" watchObservedRunningTime="2026-10-30 21:10:42.583589444 +0000 UTC m=+629.704663199" Oct 30 21:10:44.466723 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:44.466690 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:10:45.548072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:45.548018 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:45.566012 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:45.565987 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:45.577644 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:45.577616 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:10:51.297739 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.297707 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg"] Oct 30 21:10:51.303471 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.303450 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:10:51.319788 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.319761 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg"] Oct 30 21:10:51.321973 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.321946 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-tmp-dir\") pod \"premium-w1-56ac74-kserve-54d4647957-4lcpg\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:10:51.322152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.322000 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b30160-e47f-45ea-8d4c-c6450c1e86bd-tls-certs\") pod \"premium-w1-56ac74-kserve-54d4647957-4lcpg\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:10:51.322210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.322159 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-dshm\") pod \"premium-w1-56ac74-kserve-54d4647957-4lcpg\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:10:51.322252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.322207 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-model-cache\") pod \"premium-w1-56ac74-kserve-54d4647957-4lcpg\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:10:51.322304 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.322285 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4k54\" (UniqueName: \"kubernetes.io/projected/81b30160-e47f-45ea-8d4c-c6450c1e86bd-kube-api-access-p4k54\") pod \"premium-w1-56ac74-kserve-54d4647957-4lcpg\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:10:51.322371 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.322355 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-home\") pod \"premium-w1-56ac74-kserve-54d4647957-4lcpg\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:10:51.327698 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.327675 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-56ac74\"/\"premium-w1-56ac74-kserve-self-signed-certs\"" Oct 30 21:10:51.423044 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.422999 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-model-cache\") pod \"premium-w1-56ac74-kserve-54d4647957-4lcpg\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:10:51.423227 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.423107 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4k54\" (UniqueName: \"kubernetes.io/projected/81b30160-e47f-45ea-8d4c-c6450c1e86bd-kube-api-access-p4k54\") pod \"premium-w1-56ac74-kserve-54d4647957-4lcpg\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:10:51.423227 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.423165 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-home\") pod \"premium-w1-56ac74-kserve-54d4647957-4lcpg\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:10:51.423344 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.423228 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-tmp-dir\") pod \"premium-w1-56ac74-kserve-54d4647957-4lcpg\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:10:51.423344 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.423279 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b30160-e47f-45ea-8d4c-c6450c1e86bd-tls-certs\") pod \"premium-w1-56ac74-kserve-54d4647957-4lcpg\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:10:51.423451 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.423426 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-model-cache\") pod \"premium-w1-56ac74-kserve-54d4647957-4lcpg\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:10:51.423506 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.423445 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-dshm\") pod \"premium-w1-56ac74-kserve-54d4647957-4lcpg\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:10:51.423630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.423608 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-tmp-dir\") pod \"premium-w1-56ac74-kserve-54d4647957-4lcpg\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:10:51.423704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.423612 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-home\") pod \"premium-w1-56ac74-kserve-54d4647957-4lcpg\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:10:51.425574 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.425547 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-dshm\") pod \"premium-w1-56ac74-kserve-54d4647957-4lcpg\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:10:51.425948 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.425930 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b30160-e47f-45ea-8d4c-c6450c1e86bd-tls-certs\") pod \"premium-w1-56ac74-kserve-54d4647957-4lcpg\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:10:51.446812 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.446785 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4k54\" (UniqueName: \"kubernetes.io/projected/81b30160-e47f-45ea-8d4c-c6450c1e86bd-kube-api-access-p4k54\") pod \"premium-w1-56ac74-kserve-54d4647957-4lcpg\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:10:51.579402 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.579322 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:51.593141 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.593109 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:51.619272 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.619242 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:10:51.619440 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.619389 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:10:51.934916 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:51.934893 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg"] Oct 30 21:10:51.937361 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:10:51.937330 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81b30160_e47f_45ea_8d4c_c6450c1e86bd.slice/crio-fa1ab83fc155845a5bdfecdb923f61888ea9a3968db0361fb1bafd2028a4be4b WatchSource:0}: Error finding container fa1ab83fc155845a5bdfecdb923f61888ea9a3968db0361fb1bafd2028a4be4b: Status 404 returned error can't find the container with id fa1ab83fc155845a5bdfecdb923f61888ea9a3968db0361fb1bafd2028a4be4b Oct 30 21:10:52.613651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:52.613609 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" event={"ID":"81b30160-e47f-45ea-8d4c-c6450c1e86bd","Type":"ContainerStarted","Data":"9ea6e4b39b51bee1c5f01f2ad7c1d130ecf5239b88bcb5ddb6b81d970e21a267"} Oct 30 21:10:52.614189 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:52.613659 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" event={"ID":"81b30160-e47f-45ea-8d4c-c6450c1e86bd","Type":"ContainerStarted","Data":"fa1ab83fc155845a5bdfecdb923f61888ea9a3968db0361fb1bafd2028a4be4b"} Oct 30 21:10:52.619773 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:52.619738 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:10:52.652103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:52.652049 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podStartSLOduration=1.6520193970000001 podStartE2EDuration="1.652019397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:10:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:10:52.65018958 +0000 UTC m=+639.771263327" watchObservedRunningTime="2026-10-30 21:10:52.652019397 +0000 UTC m=+639.773093144" Oct 30 21:10:58.734189 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:10:58.734141 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:11:01.620436 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:01.620397 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:11:01.633878 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:01.633850 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:11:01.681349 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:01.681321 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:11:07.485141 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:11:07.485096 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:11:11.187218 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.187184 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l"] Oct 30 21:11:11.191890 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.191869 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:11.213933 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.213899 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l"] Oct 30 21:11:11.221289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.221258 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-79477e\"/\"premium-w3-79477e-kserve-self-signed-certs\"" Oct 30 21:11:11.330438 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.330396 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-model-cache\") pod \"premium-w3-79477e-kserve-bb884988d-6v96l\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:11.330438 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.330439 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-home\") pod \"premium-w3-79477e-kserve-bb884988d-6v96l\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:11.330737 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.330537 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-dshm\") pod \"premium-w3-79477e-kserve-bb884988d-6v96l\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:11.330737 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.330574 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-tmp-dir\") pod \"premium-w3-79477e-kserve-bb884988d-6v96l\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:11.330737 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.330611 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rctr\" (UniqueName: \"kubernetes.io/projected/cd3973a3-874a-4e67-a809-60b41b853435-kube-api-access-9rctr\") pod \"premium-w3-79477e-kserve-bb884988d-6v96l\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:11.330737 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.330701 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd3973a3-874a-4e67-a809-60b41b853435-tls-certs\") pod \"premium-w3-79477e-kserve-bb884988d-6v96l\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:11.432315 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.432284 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-tmp-dir\") pod \"premium-w3-79477e-kserve-bb884988d-6v96l\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:11.432512 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.432333 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rctr\" (UniqueName: \"kubernetes.io/projected/cd3973a3-874a-4e67-a809-60b41b853435-kube-api-access-9rctr\") pod \"premium-w3-79477e-kserve-bb884988d-6v96l\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:11.432512 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.432376 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd3973a3-874a-4e67-a809-60b41b853435-tls-certs\") pod \"premium-w3-79477e-kserve-bb884988d-6v96l\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:11.432512 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.432435 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-model-cache\") pod \"premium-w3-79477e-kserve-bb884988d-6v96l\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:11.432512 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.432463 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-home\") pod \"premium-w3-79477e-kserve-bb884988d-6v96l\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:11.432512 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.432490 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-dshm\") pod \"premium-w3-79477e-kserve-bb884988d-6v96l\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:11.432806 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.432780 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-tmp-dir\") pod \"premium-w3-79477e-kserve-bb884988d-6v96l\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:11.433412 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.433386 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-model-cache\") pod \"premium-w3-79477e-kserve-bb884988d-6v96l\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:11.433624 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.433451 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-home\") pod \"premium-w3-79477e-kserve-bb884988d-6v96l\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:11.434842 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.434816 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-dshm\") pod \"premium-w3-79477e-kserve-bb884988d-6v96l\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:11.435221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.435202 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd3973a3-874a-4e67-a809-60b41b853435-tls-certs\") pod \"premium-w3-79477e-kserve-bb884988d-6v96l\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:11.457724 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.457665 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rctr\" (UniqueName: \"kubernetes.io/projected/cd3973a3-874a-4e67-a809-60b41b853435-kube-api-access-9rctr\") pod \"premium-w3-79477e-kserve-bb884988d-6v96l\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:11.507212 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.507184 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:11.804847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:11.804798 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l"] Oct 30 21:11:11.807333 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:11:11.807308 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd3973a3_874a_4e67_a809_60b41b853435.slice/crio-d3e99b87ba75ba8f4df1c762b69714e809c5ad088d5f266a7e4276883c940342 WatchSource:0}: Error finding container d3e99b87ba75ba8f4df1c762b69714e809c5ad088d5f266a7e4276883c940342: Status 404 returned error can't find the container with id d3e99b87ba75ba8f4df1c762b69714e809c5ad088d5f266a7e4276883c940342 Oct 30 21:11:12.739695 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:12.739654 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" event={"ID":"cd3973a3-874a-4e67-a809-60b41b853435","Type":"ContainerStarted","Data":"42dc88d0b40943b1564b022a0be121c14617227799ecb0751d2e296433e0d279"} Oct 30 21:11:12.740097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:12.739703 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" event={"ID":"cd3973a3-874a-4e67-a809-60b41b853435","Type":"ContainerStarted","Data":"d3e99b87ba75ba8f4df1c762b69714e809c5ad088d5f266a7e4276883c940342"} Oct 30 21:11:12.780871 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:12.780825 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podStartSLOduration=1.780810907 podStartE2EDuration="1.780810907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:12.779969945 +0000 UTC m=+659.901043704" watchObservedRunningTime="2026-10-30 21:11:12.780810907 +0000 UTC m=+659.901884654" Oct 30 21:11:13.507452 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:13.507421 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:15.144738 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:15.144703 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:11:21.507302 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:21.507259 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:21.520042 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:21.520007 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:21.807529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:21.807451 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:11:23.369072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.366003 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx"] Oct 30 21:11:23.371910 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.371887 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:23.385022 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.384994 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx"] Oct 30 21:11:23.392991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.392969 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ed088d\"/\"premium-w6-ed088d-kserve-self-signed-certs\"" Oct 30 21:11:23.459531 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.459499 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-home\") pod \"premium-w6-ed088d-kserve-7669484547-xjwkx\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:23.459712 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.459626 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-dshm\") pod \"premium-w6-ed088d-kserve-7669484547-xjwkx\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:23.459712 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.459664 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-tls-certs\") pod \"premium-w6-ed088d-kserve-7669484547-xjwkx\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:23.459816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.459789 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmk66\" (UniqueName: \"kubernetes.io/projected/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-kube-api-access-cmk66\") pod \"premium-w6-ed088d-kserve-7669484547-xjwkx\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:23.459869 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.459854 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-tmp-dir\") pod \"premium-w6-ed088d-kserve-7669484547-xjwkx\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:23.459955 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.459940 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-model-cache\") pod \"premium-w6-ed088d-kserve-7669484547-xjwkx\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:23.561614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.561574 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-dshm\") pod \"premium-w6-ed088d-kserve-7669484547-xjwkx\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:23.562396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.562363 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-tls-certs\") pod \"premium-w6-ed088d-kserve-7669484547-xjwkx\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:23.562531 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.562435 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmk66\" (UniqueName: \"kubernetes.io/projected/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-kube-api-access-cmk66\") pod \"premium-w6-ed088d-kserve-7669484547-xjwkx\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:23.562531 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.562483 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-tmp-dir\") pod \"premium-w6-ed088d-kserve-7669484547-xjwkx\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:23.562531 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.562526 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-model-cache\") pod \"premium-w6-ed088d-kserve-7669484547-xjwkx\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:23.562690 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.562607 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-home\") pod \"premium-w6-ed088d-kserve-7669484547-xjwkx\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:23.563009 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.562989 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-home\") pod \"premium-w6-ed088d-kserve-7669484547-xjwkx\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:23.564332 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.563616 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-tmp-dir\") pod \"premium-w6-ed088d-kserve-7669484547-xjwkx\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:23.564332 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.564252 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-model-cache\") pod \"premium-w6-ed088d-kserve-7669484547-xjwkx\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:23.571326 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.571300 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-dshm\") pod \"premium-w6-ed088d-kserve-7669484547-xjwkx\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:23.572119 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.572097 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-tls-certs\") pod \"premium-w6-ed088d-kserve-7669484547-xjwkx\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:23.587756 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.587724 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmk66\" (UniqueName: \"kubernetes.io/projected/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-kube-api-access-cmk66\") pod \"premium-w6-ed088d-kserve-7669484547-xjwkx\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:23.694708 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:23.694670 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:24.002350 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:24.002274 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx"] Oct 30 21:11:24.007389 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:11:24.007363 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cb8157_aaea_41e8_af1a_aef07b8a9bc4.slice/crio-54fa72b8f73aae25edf1568649be580a142d56580d6a6b5846b3bc237781fec4 WatchSource:0}: Error finding container 54fa72b8f73aae25edf1568649be580a142d56580d6a6b5846b3bc237781fec4: Status 404 returned error can't find the container with id 54fa72b8f73aae25edf1568649be580a142d56580d6a6b5846b3bc237781fec4 Oct 30 21:11:24.818803 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:24.818711 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" event={"ID":"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4","Type":"ContainerStarted","Data":"e035d50d5ae960a3ed6d8881755e550630c1ac4feb05d2fe90bd99c989eeab9a"} Oct 30 21:11:24.818803 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:24.818757 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" event={"ID":"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4","Type":"ContainerStarted","Data":"54fa72b8f73aae25edf1568649be580a142d56580d6a6b5846b3bc237781fec4"} Oct 30 21:11:24.858876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:24.858809 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podStartSLOduration=1.858791994 podStartE2EDuration="1.858791994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:11:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:11:24.855629432 +0000 UTC m=+671.976703180" watchObservedRunningTime="2026-10-30 21:11:24.858791994 +0000 UTC m=+671.979865743" Oct 30 21:11:25.695211 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:25.695175 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:33.695853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:33.695818 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:33.708998 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:33.708972 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:33.885360 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:33.885329 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:11:35.792528 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:11:35.792493 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:11:37.484367 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:11:37.484332 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:12:03.140367 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:03.140328 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:12:07.485572 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:12:07.485473 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:12:12.151223 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:12.151191 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl"] Oct 30 21:12:12.155086 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:12.155068 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl" Oct 30 21:12:12.169172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:12.169147 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3d6e1d2-c720-4883-a300-501d5b3847b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97367ea0-v5wdl\" (UID: \"c3d6e1d2-c720-4883-a300-501d5b3847b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl" Oct 30 21:12:12.169285 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:12.169212 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3d6e1d2-c720-4883-a300-501d5b3847b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97367ea0-v5wdl\" (UID: \"c3d6e1d2-c720-4883-a300-501d5b3847b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl" Oct 30 21:12:12.169327 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:12.169298 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z79vs\" (UniqueName: \"kubernetes.io/projected/c3d6e1d2-c720-4883-a300-501d5b3847b7-kube-api-access-z79vs\") pod \"maas-api-revoke-sub-97367ea0-v5wdl\" (UID: \"c3d6e1d2-c720-4883-a300-501d5b3847b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl" Oct 30 21:12:12.183623 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:12.183600 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:12:12.183792 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:12.183701 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:12:12.186138 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:12.186118 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl"] Oct 30 21:12:12.270825 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:12.270789 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3d6e1d2-c720-4883-a300-501d5b3847b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97367ea0-v5wdl\" (UID: \"c3d6e1d2-c720-4883-a300-501d5b3847b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl" Oct 30 21:12:12.270825 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:12.270834 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z79vs\" (UniqueName: \"kubernetes.io/projected/c3d6e1d2-c720-4883-a300-501d5b3847b7-kube-api-access-z79vs\") pod \"maas-api-revoke-sub-97367ea0-v5wdl\" (UID: \"c3d6e1d2-c720-4883-a300-501d5b3847b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl" Oct 30 21:12:12.271028 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:12.270906 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3d6e1d2-c720-4883-a300-501d5b3847b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97367ea0-v5wdl\" (UID: \"c3d6e1d2-c720-4883-a300-501d5b3847b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl" Oct 30 21:12:12.271499 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:12.271472 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3d6e1d2-c720-4883-a300-501d5b3847b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97367ea0-v5wdl\" (UID: \"c3d6e1d2-c720-4883-a300-501d5b3847b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl" Oct 30 21:12:12.273282 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:12.273257 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3d6e1d2-c720-4883-a300-501d5b3847b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97367ea0-v5wdl\" (UID: \"c3d6e1d2-c720-4883-a300-501d5b3847b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl" Oct 30 21:12:12.291607 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:12.291516 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z79vs\" (UniqueName: \"kubernetes.io/projected/c3d6e1d2-c720-4883-a300-501d5b3847b7-kube-api-access-z79vs\") pod \"maas-api-revoke-sub-97367ea0-v5wdl\" (UID: \"c3d6e1d2-c720-4883-a300-501d5b3847b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl" Oct 30 21:12:12.469576 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:12.469544 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl" Oct 30 21:12:12.776963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:12.776937 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl"] Oct 30 21:12:12.779239 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:12:12.779211 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3d6e1d2_c720_4883_a300_501d5b3847b7.slice/crio-ee28b7e932048b77fb792ea02755a8181421ead1e6a5586931be6231c48af002 WatchSource:0}: Error finding container ee28b7e932048b77fb792ea02755a8181421ead1e6a5586931be6231c48af002: Status 404 returned error can't find the container with id ee28b7e932048b77fb792ea02755a8181421ead1e6a5586931be6231c48af002 Oct 30 21:12:12.781674 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:12.781649 4926 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:12:13.111017 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:13.110943 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl" event={"ID":"c3d6e1d2-c720-4883-a300-501d5b3847b7","Type":"ContainerStarted","Data":"ee28b7e932048b77fb792ea02755a8181421ead1e6a5586931be6231c48af002"} Oct 30 21:12:15.126212 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:15.126176 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl" event={"ID":"c3d6e1d2-c720-4883-a300-501d5b3847b7","Type":"ContainerStarted","Data":"7e01242e5029a737877f4ba2728d5cd62340f92bf4faeee712a8e3994014ec07"} Oct 30 21:12:15.170396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:15.170336 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl" podStartSLOduration=1.3456452429999999 podStartE2EDuration="3.170319276s" totalImagesPullingTime="1.824674033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:15.168113225 +0000 UTC m=+722.289186968" watchObservedRunningTime="2026-10-30 21:12:15.170319276 +0000 UTC m=+722.291393023" Oct 30 21:12:16.134258 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:16.134227 4926 generic.go:356] "Generic (PLEG): container finished" podID="c3d6e1d2-c720-4883-a300-501d5b3847b7" containerID="7e01242e5029a737877f4ba2728d5cd62340f92bf4faeee712a8e3994014ec07" exitCode=0 Oct 30 21:12:16.134654 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:16.134315 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl" event={"ID":"c3d6e1d2-c720-4883-a300-501d5b3847b7","Type":"ContainerDied","Data":"7e01242e5029a737877f4ba2728d5cd62340f92bf4faeee712a8e3994014ec07"} Oct 30 21:12:17.290704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:17.290676 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl" Oct 30 21:12:17.331043 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:17.331009 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3d6e1d2-c720-4883-a300-501d5b3847b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3d6e1d2-c720-4883-a300-501d5b3847b7-maas-api-xks-ca\") pod \"c3d6e1d2-c720-4883-a300-501d5b3847b7\" (UID: \"c3d6e1d2-c720-4883-a300-501d5b3847b7\") " Oct 30 21:12:17.331190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:17.331078 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3d6e1d2-c720-4883-a300-501d5b3847b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3d6e1d2-c720-4883-a300-501d5b3847b7-maas-api-openshift-ca\") pod \"c3d6e1d2-c720-4883-a300-501d5b3847b7\" (UID: \"c3d6e1d2-c720-4883-a300-501d5b3847b7\") " Oct 30 21:12:17.331190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:17.331144 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3d6e1d2-c720-4883-a300-501d5b3847b7-kube-api-access-z79vs\" (UniqueName: \"kubernetes.io/projected/c3d6e1d2-c720-4883-a300-501d5b3847b7-kube-api-access-z79vs\") pod \"c3d6e1d2-c720-4883-a300-501d5b3847b7\" (UID: \"c3d6e1d2-c720-4883-a300-501d5b3847b7\") " Oct 30 21:12:17.331450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:17.331424 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3d6e1d2-c720-4883-a300-501d5b3847b7-maas-api-openshift-ca" pod "c3d6e1d2-c720-4883-a300-501d5b3847b7" (UID: "c3d6e1d2-c720-4883-a300-501d5b3847b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:12:17.332090 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:17.331562 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3d6e1d2-c720-4883-a300-501d5b3847b7-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:12:17.333385 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:17.333361 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3d6e1d2-c720-4883-a300-501d5b3847b7-kube-api-access-z79vs" pod "c3d6e1d2-c720-4883-a300-501d5b3847b7" (UID: "c3d6e1d2-c720-4883-a300-501d5b3847b7"). InnerVolumeSpecName "kube-api-access-z79vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:12:17.333500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:17.333388 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3d6e1d2-c720-4883-a300-501d5b3847b7-maas-api-xks-ca" pod "c3d6e1d2-c720-4883-a300-501d5b3847b7" (UID: "c3d6e1d2-c720-4883-a300-501d5b3847b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:12:17.432962 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:17.432935 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3d6e1d2-c720-4883-a300-501d5b3847b7-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:12:17.432962 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:17.432959 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z79vs\" (UniqueName: \"kubernetes.io/projected/c3d6e1d2-c720-4883-a300-501d5b3847b7-kube-api-access-z79vs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:12:18.146397 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:18.146366 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl" Oct 30 21:12:18.148479 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:18.146394 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl" event={"ID":"c3d6e1d2-c720-4883-a300-501d5b3847b7","Type":"ContainerDied","Data":"ee28b7e932048b77fb792ea02755a8181421ead1e6a5586931be6231c48af002"} Oct 30 21:12:18.148479 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:18.146452 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee28b7e932048b77fb792ea02755a8181421ead1e6a5586931be6231c48af002" Oct 30 21:12:27.560920 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:27.560885 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04757449-2s5fd"] Oct 30 21:12:27.561407 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:27.561391 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c3d6e1d2-c720-4883-a300-501d5b3847b7" containerName="revoke-keys" Oct 30 21:12:27.561477 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:27.561411 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3d6e1d2-c720-4883-a300-501d5b3847b7" containerName="revoke-keys" Oct 30 21:12:27.561520 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:27.561514 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3d6e1d2-c720-4883-a300-501d5b3847b7" containerName="revoke-keys" Oct 30 21:12:27.564537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:27.564522 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04757449-2s5fd" Oct 30 21:12:27.583120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:27.583089 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04757449-2s5fd"] Oct 30 21:12:27.589573 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:27.589549 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:12:27.589749 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:27.589621 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:12:27.635949 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:27.635922 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rltt\" (UniqueName: \"kubernetes.io/projected/84b3c1f7-f3b0-4f7b-8359-96002c9840fb-kube-api-access-7rltt\") pod \"maas-api-revoke-sub-04757449-2s5fd\" (UID: \"84b3c1f7-f3b0-4f7b-8359-96002c9840fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04757449-2s5fd" Oct 30 21:12:27.636114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:27.636076 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b3c1f7-f3b0-4f7b-8359-96002c9840fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04757449-2s5fd\" (UID: \"84b3c1f7-f3b0-4f7b-8359-96002c9840fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04757449-2s5fd" Oct 30 21:12:27.636193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:27.636177 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b3c1f7-f3b0-4f7b-8359-96002c9840fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04757449-2s5fd\" (UID: \"84b3c1f7-f3b0-4f7b-8359-96002c9840fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04757449-2s5fd" Oct 30 21:12:27.737170 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:27.737132 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rltt\" (UniqueName: \"kubernetes.io/projected/84b3c1f7-f3b0-4f7b-8359-96002c9840fb-kube-api-access-7rltt\") pod \"maas-api-revoke-sub-04757449-2s5fd\" (UID: \"84b3c1f7-f3b0-4f7b-8359-96002c9840fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04757449-2s5fd" Oct 30 21:12:27.737366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:27.737220 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b3c1f7-f3b0-4f7b-8359-96002c9840fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04757449-2s5fd\" (UID: \"84b3c1f7-f3b0-4f7b-8359-96002c9840fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04757449-2s5fd" Oct 30 21:12:27.737366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:27.737287 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b3c1f7-f3b0-4f7b-8359-96002c9840fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04757449-2s5fd\" (UID: \"84b3c1f7-f3b0-4f7b-8359-96002c9840fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04757449-2s5fd" Oct 30 21:12:27.737878 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:27.737850 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b3c1f7-f3b0-4f7b-8359-96002c9840fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04757449-2s5fd\" (UID: \"84b3c1f7-f3b0-4f7b-8359-96002c9840fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04757449-2s5fd" Oct 30 21:12:27.739682 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:27.739653 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b3c1f7-f3b0-4f7b-8359-96002c9840fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04757449-2s5fd\" (UID: \"84b3c1f7-f3b0-4f7b-8359-96002c9840fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04757449-2s5fd" Oct 30 21:12:27.755308 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:27.755285 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rltt\" (UniqueName: \"kubernetes.io/projected/84b3c1f7-f3b0-4f7b-8359-96002c9840fb-kube-api-access-7rltt\") pod \"maas-api-revoke-sub-04757449-2s5fd\" (UID: \"84b3c1f7-f3b0-4f7b-8359-96002c9840fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04757449-2s5fd" Oct 30 21:12:27.879978 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:27.879903 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04757449-2s5fd" Oct 30 21:12:28.176274 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:28.176249 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04757449-2s5fd"] Oct 30 21:12:28.179569 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:12:28.179526 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b3c1f7_f3b0_4f7b_8359_96002c9840fb.slice/crio-8a81ecbfc03a79784b1e21107015b11671c295ee0e62ff72d6fcbf6405b3c8eb WatchSource:0}: Error finding container 8a81ecbfc03a79784b1e21107015b11671c295ee0e62ff72d6fcbf6405b3c8eb: Status 404 returned error can't find the container with id 8a81ecbfc03a79784b1e21107015b11671c295ee0e62ff72d6fcbf6405b3c8eb Oct 30 21:12:28.216484 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:28.216454 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04757449-2s5fd" event={"ID":"84b3c1f7-f3b0-4f7b-8359-96002c9840fb","Type":"ContainerStarted","Data":"8a81ecbfc03a79784b1e21107015b11671c295ee0e62ff72d6fcbf6405b3c8eb"} Oct 30 21:12:29.224050 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:29.224007 4926 generic.go:356] "Generic (PLEG): container finished" podID="84b3c1f7-f3b0-4f7b-8359-96002c9840fb" containerID="f21b61e0c463b08366fd6c01f7b0e1645a0180405081a805b9b3f894d3ced0ac" exitCode=0 Oct 30 21:12:29.224498 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:29.224093 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04757449-2s5fd" event={"ID":"84b3c1f7-f3b0-4f7b-8359-96002c9840fb","Type":"ContainerDied","Data":"f21b61e0c463b08366fd6c01f7b0e1645a0180405081a805b9b3f894d3ced0ac"} Oct 30 21:12:29.585502 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:29.585471 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-660e8d66-l5s9r"] Oct 30 21:12:29.589326 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:29.589308 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660e8d66-l5s9r" Oct 30 21:12:29.607777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:29.607751 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-660e8d66-l5s9r"] Oct 30 21:12:29.664346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:29.664315 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f419ca1-8aa5-449e-bde2-1d3c6de70c84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-660e8d66-l5s9r\" (UID: \"6f419ca1-8aa5-449e-bde2-1d3c6de70c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660e8d66-l5s9r" Oct 30 21:12:29.664476 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:29.664371 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f419ca1-8aa5-449e-bde2-1d3c6de70c84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-660e8d66-l5s9r\" (UID: \"6f419ca1-8aa5-449e-bde2-1d3c6de70c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660e8d66-l5s9r" Oct 30 21:12:29.664552 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:29.664531 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpbbl\" (UniqueName: \"kubernetes.io/projected/6f419ca1-8aa5-449e-bde2-1d3c6de70c84-kube-api-access-cpbbl\") pod \"maas-api-revoke-sub-660e8d66-l5s9r\" (UID: \"6f419ca1-8aa5-449e-bde2-1d3c6de70c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660e8d66-l5s9r" Oct 30 21:12:29.765261 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:29.765231 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpbbl\" (UniqueName: \"kubernetes.io/projected/6f419ca1-8aa5-449e-bde2-1d3c6de70c84-kube-api-access-cpbbl\") pod \"maas-api-revoke-sub-660e8d66-l5s9r\" (UID: \"6f419ca1-8aa5-449e-bde2-1d3c6de70c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660e8d66-l5s9r" Oct 30 21:12:29.765462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:29.765318 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f419ca1-8aa5-449e-bde2-1d3c6de70c84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-660e8d66-l5s9r\" (UID: \"6f419ca1-8aa5-449e-bde2-1d3c6de70c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660e8d66-l5s9r" Oct 30 21:12:29.765462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:29.765357 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f419ca1-8aa5-449e-bde2-1d3c6de70c84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-660e8d66-l5s9r\" (UID: \"6f419ca1-8aa5-449e-bde2-1d3c6de70c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660e8d66-l5s9r" Oct 30 21:12:29.766156 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:29.766125 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f419ca1-8aa5-449e-bde2-1d3c6de70c84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-660e8d66-l5s9r\" (UID: \"6f419ca1-8aa5-449e-bde2-1d3c6de70c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660e8d66-l5s9r" Oct 30 21:12:29.767933 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:29.767907 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f419ca1-8aa5-449e-bde2-1d3c6de70c84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-660e8d66-l5s9r\" (UID: \"6f419ca1-8aa5-449e-bde2-1d3c6de70c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660e8d66-l5s9r" Oct 30 21:12:29.810639 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:29.810612 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpbbl\" (UniqueName: \"kubernetes.io/projected/6f419ca1-8aa5-449e-bde2-1d3c6de70c84-kube-api-access-cpbbl\") pod \"maas-api-revoke-sub-660e8d66-l5s9r\" (UID: \"6f419ca1-8aa5-449e-bde2-1d3c6de70c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-660e8d66-l5s9r" Oct 30 21:12:29.902728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:29.902653 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660e8d66-l5s9r" Oct 30 21:12:30.209474 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:30.209441 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-660e8d66-l5s9r"] Oct 30 21:12:30.212904 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:12:30.212863 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f419ca1_8aa5_449e_bde2_1d3c6de70c84.slice/crio-36a549b0ba6e4e4ed0372e595062b19e874009f69e09f27d6435a9e1371e5d06 WatchSource:0}: Error finding container 36a549b0ba6e4e4ed0372e595062b19e874009f69e09f27d6435a9e1371e5d06: Status 404 returned error can't find the container with id 36a549b0ba6e4e4ed0372e595062b19e874009f69e09f27d6435a9e1371e5d06 Oct 30 21:12:30.232740 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:30.232709 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660e8d66-l5s9r" event={"ID":"6f419ca1-8aa5-449e-bde2-1d3c6de70c84","Type":"ContainerStarted","Data":"36a549b0ba6e4e4ed0372e595062b19e874009f69e09f27d6435a9e1371e5d06"} Oct 30 21:12:30.394329 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:30.394302 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04757449-2s5fd" Oct 30 21:12:30.472193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:30.472165 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b3c1f7-f3b0-4f7b-8359-96002c9840fb-kube-api-access-7rltt\" (UniqueName: \"kubernetes.io/projected/84b3c1f7-f3b0-4f7b-8359-96002c9840fb-kube-api-access-7rltt\") pod \"84b3c1f7-f3b0-4f7b-8359-96002c9840fb\" (UID: \"84b3c1f7-f3b0-4f7b-8359-96002c9840fb\") " Oct 30 21:12:30.472355 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:30.472299 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84b3c1f7-f3b0-4f7b-8359-96002c9840fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b3c1f7-f3b0-4f7b-8359-96002c9840fb-maas-api-xks-ca\") pod \"84b3c1f7-f3b0-4f7b-8359-96002c9840fb\" (UID: \"84b3c1f7-f3b0-4f7b-8359-96002c9840fb\") " Oct 30 21:12:30.472426 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:30.472405 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b3c1f7-f3b0-4f7b-8359-96002c9840fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b3c1f7-f3b0-4f7b-8359-96002c9840fb-maas-api-openshift-ca\") pod \"84b3c1f7-f3b0-4f7b-8359-96002c9840fb\" (UID: \"84b3c1f7-f3b0-4f7b-8359-96002c9840fb\") " Oct 30 21:12:30.473257 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:30.473229 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b3c1f7-f3b0-4f7b-8359-96002c9840fb-maas-api-openshift-ca" pod "84b3c1f7-f3b0-4f7b-8359-96002c9840fb" (UID: "84b3c1f7-f3b0-4f7b-8359-96002c9840fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:12:30.474896 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:30.474838 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b3c1f7-f3b0-4f7b-8359-96002c9840fb-kube-api-access-7rltt" pod "84b3c1f7-f3b0-4f7b-8359-96002c9840fb" (UID: "84b3c1f7-f3b0-4f7b-8359-96002c9840fb"). InnerVolumeSpecName "kube-api-access-7rltt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:12:30.475355 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:30.475240 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b3c1f7-f3b0-4f7b-8359-96002c9840fb-maas-api-xks-ca" pod "84b3c1f7-f3b0-4f7b-8359-96002c9840fb" (UID: "84b3c1f7-f3b0-4f7b-8359-96002c9840fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:12:30.574202 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:30.574174 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rltt\" (UniqueName: \"kubernetes.io/projected/84b3c1f7-f3b0-4f7b-8359-96002c9840fb-kube-api-access-7rltt\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:12:30.574202 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:30.574198 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84b3c1f7-f3b0-4f7b-8359-96002c9840fb-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:12:30.574202 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:30.574209 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84b3c1f7-f3b0-4f7b-8359-96002c9840fb-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:12:31.239520 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:31.239493 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04757449-2s5fd" Oct 30 21:12:31.239996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:31.239531 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04757449-2s5fd" event={"ID":"84b3c1f7-f3b0-4f7b-8359-96002c9840fb","Type":"ContainerDied","Data":"8a81ecbfc03a79784b1e21107015b11671c295ee0e62ff72d6fcbf6405b3c8eb"} Oct 30 21:12:31.239996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:31.239582 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a81ecbfc03a79784b1e21107015b11671c295ee0e62ff72d6fcbf6405b3c8eb" Oct 30 21:12:31.241162 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:31.241133 4926 generic.go:356] "Generic (PLEG): container finished" podID="6f419ca1-8aa5-449e-bde2-1d3c6de70c84" containerID="97d0a7567d2e36c866d3d074db67e3281df1f0261e88f098daed2e27219d23c4" exitCode=0 Oct 30 21:12:31.241286 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:31.241205 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660e8d66-l5s9r" event={"ID":"6f419ca1-8aa5-449e-bde2-1d3c6de70c84","Type":"ContainerDied","Data":"97d0a7567d2e36c866d3d074db67e3281df1f0261e88f098daed2e27219d23c4"} Oct 30 21:12:32.389364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:32.389338 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660e8d66-l5s9r" Oct 30 21:12:32.495051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:32.495001 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f419ca1-8aa5-449e-bde2-1d3c6de70c84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f419ca1-8aa5-449e-bde2-1d3c6de70c84-maas-api-openshift-ca\") pod \"6f419ca1-8aa5-449e-bde2-1d3c6de70c84\" (UID: \"6f419ca1-8aa5-449e-bde2-1d3c6de70c84\") " Oct 30 21:12:32.495204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:32.495115 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f419ca1-8aa5-449e-bde2-1d3c6de70c84-kube-api-access-cpbbl\" (UniqueName: \"kubernetes.io/projected/6f419ca1-8aa5-449e-bde2-1d3c6de70c84-kube-api-access-cpbbl\") pod \"6f419ca1-8aa5-449e-bde2-1d3c6de70c84\" (UID: \"6f419ca1-8aa5-449e-bde2-1d3c6de70c84\") " Oct 30 21:12:32.495263 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:32.495228 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f419ca1-8aa5-449e-bde2-1d3c6de70c84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f419ca1-8aa5-449e-bde2-1d3c6de70c84-maas-api-xks-ca\") pod \"6f419ca1-8aa5-449e-bde2-1d3c6de70c84\" (UID: \"6f419ca1-8aa5-449e-bde2-1d3c6de70c84\") " Oct 30 21:12:32.495348 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:32.495325 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f419ca1-8aa5-449e-bde2-1d3c6de70c84-maas-api-openshift-ca" pod "6f419ca1-8aa5-449e-bde2-1d3c6de70c84" (UID: "6f419ca1-8aa5-449e-bde2-1d3c6de70c84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:12:32.495651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:32.495631 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f419ca1-8aa5-449e-bde2-1d3c6de70c84-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:12:32.497329 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:32.497297 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f419ca1-8aa5-449e-bde2-1d3c6de70c84-maas-api-xks-ca" pod "6f419ca1-8aa5-449e-bde2-1d3c6de70c84" (UID: "6f419ca1-8aa5-449e-bde2-1d3c6de70c84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:12:32.497329 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:32.497310 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f419ca1-8aa5-449e-bde2-1d3c6de70c84-kube-api-access-cpbbl" pod "6f419ca1-8aa5-449e-bde2-1d3c6de70c84" (UID: "6f419ca1-8aa5-449e-bde2-1d3c6de70c84"). InnerVolumeSpecName "kube-api-access-cpbbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:12:32.596395 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:32.596365 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f419ca1-8aa5-449e-bde2-1d3c6de70c84-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:12:32.596395 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:32.596391 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpbbl\" (UniqueName: \"kubernetes.io/projected/6f419ca1-8aa5-449e-bde2-1d3c6de70c84-kube-api-access-cpbbl\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:12:33.192152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.192112 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth"] Oct 30 21:12:33.192558 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.192537 4926 cpu_manager.go:414] "containerMap: removing container" podUID="84b3c1f7-f3b0-4f7b-8359-96002c9840fb" containerName="revoke-keys" Oct 30 21:12:33.192558 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.192552 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b3c1f7-f3b0-4f7b-8359-96002c9840fb" containerName="revoke-keys" Oct 30 21:12:33.192558 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.192582 4926 cpu_manager.go:414] "containerMap: removing container" podUID="6f419ca1-8aa5-449e-bde2-1d3c6de70c84" containerName="revoke-keys" Oct 30 21:12:33.192777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.192588 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f419ca1-8aa5-449e-bde2-1d3c6de70c84" containerName="revoke-keys" Oct 30 21:12:33.192777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.192671 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f419ca1-8aa5-449e-bde2-1d3c6de70c84" containerName="revoke-keys" Oct 30 21:12:33.192777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.192688 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b3c1f7-f3b0-4f7b-8359-96002c9840fb" containerName="revoke-keys" Oct 30 21:12:33.196381 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.196363 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth" Oct 30 21:12:33.213386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.213323 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth"] Oct 30 21:12:33.255972 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.255950 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660e8d66-l5s9r" Oct 30 21:12:33.256147 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.255983 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-660e8d66-l5s9r" event={"ID":"6f419ca1-8aa5-449e-bde2-1d3c6de70c84","Type":"ContainerDied","Data":"36a549b0ba6e4e4ed0372e595062b19e874009f69e09f27d6435a9e1371e5d06"} Oct 30 21:12:33.256147 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.256043 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36a549b0ba6e4e4ed0372e595062b19e874009f69e09f27d6435a9e1371e5d06" Oct 30 21:12:33.302984 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.302908 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2d0afaf-bbfd-4383-858a-f5d78c46032a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef521603-xmvth\" (UID: \"b2d0afaf-bbfd-4383-858a-f5d78c46032a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth" Oct 30 21:12:33.303138 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.302985 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2d0afaf-bbfd-4383-858a-f5d78c46032a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef521603-xmvth\" (UID: \"b2d0afaf-bbfd-4383-858a-f5d78c46032a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth" Oct 30 21:12:33.303138 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.303074 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m57kb\" (UniqueName: \"kubernetes.io/projected/b2d0afaf-bbfd-4383-858a-f5d78c46032a-kube-api-access-m57kb\") pod \"maas-api-revoke-sub-ef521603-xmvth\" (UID: \"b2d0afaf-bbfd-4383-858a-f5d78c46032a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth" Oct 30 21:12:33.404173 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.404146 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2d0afaf-bbfd-4383-858a-f5d78c46032a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef521603-xmvth\" (UID: \"b2d0afaf-bbfd-4383-858a-f5d78c46032a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth" Oct 30 21:12:33.404630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.404196 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2d0afaf-bbfd-4383-858a-f5d78c46032a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef521603-xmvth\" (UID: \"b2d0afaf-bbfd-4383-858a-f5d78c46032a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth" Oct 30 21:12:33.404630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.404228 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m57kb\" (UniqueName: \"kubernetes.io/projected/b2d0afaf-bbfd-4383-858a-f5d78c46032a-kube-api-access-m57kb\") pod \"maas-api-revoke-sub-ef521603-xmvth\" (UID: \"b2d0afaf-bbfd-4383-858a-f5d78c46032a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth" Oct 30 21:12:33.404960 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.404938 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2d0afaf-bbfd-4383-858a-f5d78c46032a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ef521603-xmvth\" (UID: \"b2d0afaf-bbfd-4383-858a-f5d78c46032a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth" Oct 30 21:12:33.406659 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.406634 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2d0afaf-bbfd-4383-858a-f5d78c46032a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ef521603-xmvth\" (UID: \"b2d0afaf-bbfd-4383-858a-f5d78c46032a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth" Oct 30 21:12:33.423214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.423188 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m57kb\" (UniqueName: \"kubernetes.io/projected/b2d0afaf-bbfd-4383-858a-f5d78c46032a-kube-api-access-m57kb\") pod \"maas-api-revoke-sub-ef521603-xmvth\" (UID: \"b2d0afaf-bbfd-4383-858a-f5d78c46032a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth" Oct 30 21:12:33.512712 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.512684 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth" Oct 30 21:12:33.814720 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:33.814606 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth"] Oct 30 21:12:33.817419 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:12:33.817355 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d0afaf_bbfd_4383_858a_f5d78c46032a.slice/crio-7676005afaa14a7ef672153831eb4f9d5cd659378a7785b219b1a32b69354377 WatchSource:0}: Error finding container 7676005afaa14a7ef672153831eb4f9d5cd659378a7785b219b1a32b69354377: Status 404 returned error can't find the container with id 7676005afaa14a7ef672153831eb4f9d5cd659378a7785b219b1a32b69354377 Oct 30 21:12:34.264013 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:34.263979 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth" event={"ID":"b2d0afaf-bbfd-4383-858a-f5d78c46032a","Type":"ContainerStarted","Data":"21744da6d7062052128964b868bcf6843baa375a0f87032e5bdd67341122a5b4"} Oct 30 21:12:34.264215 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:34.264044 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth" event={"ID":"b2d0afaf-bbfd-4383-858a-f5d78c46032a","Type":"ContainerStarted","Data":"7676005afaa14a7ef672153831eb4f9d5cd659378a7785b219b1a32b69354377"} Oct 30 21:12:34.300983 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:34.300914 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth" podStartSLOduration=1.300895306 podStartE2EDuration="1.300895306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:34.29984298 +0000 UTC m=+741.420916728" watchObservedRunningTime="2026-10-30 21:12:34.300895306 +0000 UTC m=+741.421969054" Oct 30 21:12:35.272101 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:35.272070 4926 generic.go:356] "Generic (PLEG): container finished" podID="b2d0afaf-bbfd-4383-858a-f5d78c46032a" containerID="21744da6d7062052128964b868bcf6843baa375a0f87032e5bdd67341122a5b4" exitCode=0 Oct 30 21:12:35.272488 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:35.272165 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth" event={"ID":"b2d0afaf-bbfd-4383-858a-f5d78c46032a","Type":"ContainerDied","Data":"21744da6d7062052128964b868bcf6843baa375a0f87032e5bdd67341122a5b4"} Oct 30 21:12:35.819364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:35.819330 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:12:36.440405 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:36.440374 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth" Oct 30 21:12:36.540803 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:36.540455 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2d0afaf-bbfd-4383-858a-f5d78c46032a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2d0afaf-bbfd-4383-858a-f5d78c46032a-maas-api-openshift-ca\") pod \"b2d0afaf-bbfd-4383-858a-f5d78c46032a\" (UID: \"b2d0afaf-bbfd-4383-858a-f5d78c46032a\") " Oct 30 21:12:36.540803 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:36.540532 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2d0afaf-bbfd-4383-858a-f5d78c46032a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2d0afaf-bbfd-4383-858a-f5d78c46032a-maas-api-xks-ca\") pod \"b2d0afaf-bbfd-4383-858a-f5d78c46032a\" (UID: \"b2d0afaf-bbfd-4383-858a-f5d78c46032a\") " Oct 30 21:12:36.540803 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:36.540641 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2d0afaf-bbfd-4383-858a-f5d78c46032a-kube-api-access-m57kb\" (UniqueName: \"kubernetes.io/projected/b2d0afaf-bbfd-4383-858a-f5d78c46032a-kube-api-access-m57kb\") pod \"b2d0afaf-bbfd-4383-858a-f5d78c46032a\" (UID: \"b2d0afaf-bbfd-4383-858a-f5d78c46032a\") " Oct 30 21:12:36.541722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:36.541682 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d0afaf-bbfd-4383-858a-f5d78c46032a-maas-api-openshift-ca" pod "b2d0afaf-bbfd-4383-858a-f5d78c46032a" (UID: "b2d0afaf-bbfd-4383-858a-f5d78c46032a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:12:36.544095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:36.543961 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2d0afaf-bbfd-4383-858a-f5d78c46032a-maas-api-xks-ca" pod "b2d0afaf-bbfd-4383-858a-f5d78c46032a" (UID: "b2d0afaf-bbfd-4383-858a-f5d78c46032a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:12:36.544460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:36.544436 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d0afaf-bbfd-4383-858a-f5d78c46032a-kube-api-access-m57kb" pod "b2d0afaf-bbfd-4383-858a-f5d78c46032a" (UID: "b2d0afaf-bbfd-4383-858a-f5d78c46032a"). InnerVolumeSpecName "kube-api-access-m57kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:12:36.642576 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:36.642529 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m57kb\" (UniqueName: \"kubernetes.io/projected/b2d0afaf-bbfd-4383-858a-f5d78c46032a-kube-api-access-m57kb\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:12:36.642576 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:36.642563 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2d0afaf-bbfd-4383-858a-f5d78c46032a-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:12:36.642576 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:36.642578 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2d0afaf-bbfd-4383-858a-f5d78c46032a-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:12:37.290075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:37.289946 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth" Oct 30 21:12:37.290075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:37.289958 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth" event={"ID":"b2d0afaf-bbfd-4383-858a-f5d78c46032a","Type":"ContainerDied","Data":"7676005afaa14a7ef672153831eb4f9d5cd659378a7785b219b1a32b69354377"} Oct 30 21:12:37.290075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:37.290019 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7676005afaa14a7ef672153831eb4f9d5cd659378a7785b219b1a32b69354377" Oct 30 21:12:37.528592 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:12:37.528386 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:12:40.406384 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.406352 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e8c81a-qgb9p"] Oct 30 21:12:40.406865 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.406773 4926 cpu_manager.go:414] "containerMap: removing container" podUID="b2d0afaf-bbfd-4383-858a-f5d78c46032a" containerName="revoke-keys" Oct 30 21:12:40.406865 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.406784 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2d0afaf-bbfd-4383-858a-f5d78c46032a" containerName="revoke-keys" Oct 30 21:12:40.407050 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.406886 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2d0afaf-bbfd-4383-858a-f5d78c46032a" containerName="revoke-keys" Oct 30 21:12:40.410963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.410936 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e8c81a-qgb9p" Oct 30 21:12:40.429268 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.429240 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e8c81a-qgb9p"] Oct 30 21:12:40.436142 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.436122 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:12:40.436297 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.436162 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:12:40.498239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.498206 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd710cc-405e-4c11-82b9-82e3d8ec793b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3e8c81a-qgb9p\" (UID: \"0fd710cc-405e-4c11-82b9-82e3d8ec793b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e8c81a-qgb9p" Oct 30 21:12:40.498239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.498246 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd710cc-405e-4c11-82b9-82e3d8ec793b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3e8c81a-qgb9p\" (UID: \"0fd710cc-405e-4c11-82b9-82e3d8ec793b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e8c81a-qgb9p" Oct 30 21:12:40.498483 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.498314 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fzw7\" (UniqueName: \"kubernetes.io/projected/0fd710cc-405e-4c11-82b9-82e3d8ec793b-kube-api-access-4fzw7\") pod \"maas-api-revoke-sub-b3e8c81a-qgb9p\" (UID: \"0fd710cc-405e-4c11-82b9-82e3d8ec793b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e8c81a-qgb9p" Oct 30 21:12:40.599294 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.599256 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fzw7\" (UniqueName: \"kubernetes.io/projected/0fd710cc-405e-4c11-82b9-82e3d8ec793b-kube-api-access-4fzw7\") pod \"maas-api-revoke-sub-b3e8c81a-qgb9p\" (UID: \"0fd710cc-405e-4c11-82b9-82e3d8ec793b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e8c81a-qgb9p" Oct 30 21:12:40.599494 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.599424 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd710cc-405e-4c11-82b9-82e3d8ec793b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3e8c81a-qgb9p\" (UID: \"0fd710cc-405e-4c11-82b9-82e3d8ec793b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e8c81a-qgb9p" Oct 30 21:12:40.599494 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.599464 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd710cc-405e-4c11-82b9-82e3d8ec793b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3e8c81a-qgb9p\" (UID: \"0fd710cc-405e-4c11-82b9-82e3d8ec793b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e8c81a-qgb9p" Oct 30 21:12:40.600101 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.600077 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd710cc-405e-4c11-82b9-82e3d8ec793b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3e8c81a-qgb9p\" (UID: \"0fd710cc-405e-4c11-82b9-82e3d8ec793b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e8c81a-qgb9p" Oct 30 21:12:40.601881 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.601860 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd710cc-405e-4c11-82b9-82e3d8ec793b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3e8c81a-qgb9p\" (UID: \"0fd710cc-405e-4c11-82b9-82e3d8ec793b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e8c81a-qgb9p" Oct 30 21:12:40.621968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.621941 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fzw7\" (UniqueName: \"kubernetes.io/projected/0fd710cc-405e-4c11-82b9-82e3d8ec793b-kube-api-access-4fzw7\") pod \"maas-api-revoke-sub-b3e8c81a-qgb9p\" (UID: \"0fd710cc-405e-4c11-82b9-82e3d8ec793b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e8c81a-qgb9p" Oct 30 21:12:40.725358 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.725318 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e8c81a-qgb9p" Oct 30 21:12:40.727422 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.727325 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9"] Oct 30 21:12:40.745527 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.745460 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:40.754431 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.754329 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9"] Oct 30 21:12:40.760381 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.760350 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-79477e\"/\"distinct-w3-79477e-kserve-self-signed-certs\"" Oct 30 21:12:40.803051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.802069 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-tmp-dir\") pod \"distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:40.803051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.802135 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-model-cache\") pod \"distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:40.803051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.802197 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-dshm\") pod \"distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:40.803051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.802290 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz7lv\" (UniqueName: \"kubernetes.io/projected/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-kube-api-access-wz7lv\") pod \"distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:40.803051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.802322 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-tls-certs\") pod \"distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:40.803051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.802365 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-home\") pod \"distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:40.903245 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.903214 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-dshm\") pod \"distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:40.903448 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.903295 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz7lv\" (UniqueName: \"kubernetes.io/projected/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-kube-api-access-wz7lv\") pod \"distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:40.903448 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.903318 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-tls-certs\") pod \"distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:40.903448 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.903352 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-home\") pod \"distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:40.903672 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.903647 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-tmp-dir\") pod \"distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:40.903732 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.903709 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-model-cache\") pod \"distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:40.903786 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.903759 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-home\") pod \"distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:40.904072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.904049 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-tmp-dir\") pod \"distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:40.904178 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.904086 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-model-cache\") pod \"distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:40.905918 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.905884 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-dshm\") pod \"distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:40.906084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.906064 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-tls-certs\") pod \"distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:40.922722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:40.922697 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz7lv\" (UniqueName: \"kubernetes.io/projected/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-kube-api-access-wz7lv\") pod \"distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:41.065857 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:41.065711 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e8c81a-qgb9p"] Oct 30 21:12:41.068974 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:12:41.068943 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd710cc_405e_4c11_82b9_82e3d8ec793b.slice/crio-11ecb13b78b5c19a0e4d1dc5c08932677321b6b27cfa9c2f9b7d294ab9d347b9 WatchSource:0}: Error finding container 11ecb13b78b5c19a0e4d1dc5c08932677321b6b27cfa9c2f9b7d294ab9d347b9: Status 404 returned error can't find the container with id 11ecb13b78b5c19a0e4d1dc5c08932677321b6b27cfa9c2f9b7d294ab9d347b9 Oct 30 21:12:41.084981 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:41.084956 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:41.321388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:41.321312 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e8c81a-qgb9p" event={"ID":"0fd710cc-405e-4c11-82b9-82e3d8ec793b","Type":"ContainerStarted","Data":"11ecb13b78b5c19a0e4d1dc5c08932677321b6b27cfa9c2f9b7d294ab9d347b9"} Oct 30 21:12:41.384249 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:41.384225 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9"] Oct 30 21:12:41.386696 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:12:41.386657 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6f1e2cb_36a7_435d_b61e_8ab3272c591a.slice/crio-37b5e24f69365cf60dca91eb10c6807d5775964b1a0f9c36997aabff33e05321 WatchSource:0}: Error finding container 37b5e24f69365cf60dca91eb10c6807d5775964b1a0f9c36997aabff33e05321: Status 404 returned error can't find the container with id 37b5e24f69365cf60dca91eb10c6807d5775964b1a0f9c36997aabff33e05321 Oct 30 21:12:42.329603 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:42.329561 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" event={"ID":"c6f1e2cb-36a7-435d-b61e-8ab3272c591a","Type":"ContainerStarted","Data":"db69411cdd265effbe3d89419b224b951d6f8e77a09f653110e2ffb61d69aa1f"} Oct 30 21:12:42.330052 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:42.329614 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" event={"ID":"c6f1e2cb-36a7-435d-b61e-8ab3272c591a","Type":"ContainerStarted","Data":"37b5e24f69365cf60dca91eb10c6807d5775964b1a0f9c36997aabff33e05321"} Oct 30 21:12:42.331266 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:42.331235 4926 generic.go:356] "Generic (PLEG): container finished" podID="0fd710cc-405e-4c11-82b9-82e3d8ec793b" containerID="462d008cb14df15b2d27906660a78599b32cfc9fb5b22544c48ad26b3946d1ff" exitCode=0 Oct 30 21:12:42.331375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:42.331286 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e8c81a-qgb9p" event={"ID":"0fd710cc-405e-4c11-82b9-82e3d8ec793b","Type":"ContainerDied","Data":"462d008cb14df15b2d27906660a78599b32cfc9fb5b22544c48ad26b3946d1ff"} Oct 30 21:12:42.372731 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:42.372678 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podStartSLOduration=2.3726621359999998 podStartE2EDuration="2.372662136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:42.369538047 +0000 UTC m=+749.490611794" watchObservedRunningTime="2026-10-30 21:12:42.372662136 +0000 UTC m=+749.493735882" Oct 30 21:12:43.085341 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:43.085311 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:43.498442 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:43.498415 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e8c81a-qgb9p" Oct 30 21:12:43.532505 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:43.532481 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fd710cc-405e-4c11-82b9-82e3d8ec793b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd710cc-405e-4c11-82b9-82e3d8ec793b-maas-api-xks-ca\") pod \"0fd710cc-405e-4c11-82b9-82e3d8ec793b\" (UID: \"0fd710cc-405e-4c11-82b9-82e3d8ec793b\") " Oct 30 21:12:43.532658 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:43.532640 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fd710cc-405e-4c11-82b9-82e3d8ec793b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd710cc-405e-4c11-82b9-82e3d8ec793b-maas-api-openshift-ca\") pod \"0fd710cc-405e-4c11-82b9-82e3d8ec793b\" (UID: \"0fd710cc-405e-4c11-82b9-82e3d8ec793b\") " Oct 30 21:12:43.532737 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:43.532686 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd710cc-405e-4c11-82b9-82e3d8ec793b-kube-api-access-4fzw7\" (UniqueName: \"kubernetes.io/projected/0fd710cc-405e-4c11-82b9-82e3d8ec793b-kube-api-access-4fzw7\") pod \"0fd710cc-405e-4c11-82b9-82e3d8ec793b\" (UID: \"0fd710cc-405e-4c11-82b9-82e3d8ec793b\") " Oct 30 21:12:43.532979 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:43.532947 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd710cc-405e-4c11-82b9-82e3d8ec793b-maas-api-openshift-ca" pod "0fd710cc-405e-4c11-82b9-82e3d8ec793b" (UID: "0fd710cc-405e-4c11-82b9-82e3d8ec793b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:12:43.534693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:43.534658 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd710cc-405e-4c11-82b9-82e3d8ec793b-maas-api-xks-ca" pod "0fd710cc-405e-4c11-82b9-82e3d8ec793b" (UID: "0fd710cc-405e-4c11-82b9-82e3d8ec793b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:12:43.534918 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:43.534892 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd710cc-405e-4c11-82b9-82e3d8ec793b-kube-api-access-4fzw7" pod "0fd710cc-405e-4c11-82b9-82e3d8ec793b" (UID: "0fd710cc-405e-4c11-82b9-82e3d8ec793b"). InnerVolumeSpecName "kube-api-access-4fzw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:12:43.634081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:43.633993 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fd710cc-405e-4c11-82b9-82e3d8ec793b-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:12:43.634081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:43.634020 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fzw7\" (UniqueName: \"kubernetes.io/projected/0fd710cc-405e-4c11-82b9-82e3d8ec793b-kube-api-access-4fzw7\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:12:43.634081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:43.634044 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fd710cc-405e-4c11-82b9-82e3d8ec793b-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:12:44.344902 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:44.344870 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e8c81a-qgb9p" Oct 30 21:12:44.345086 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:44.344901 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e8c81a-qgb9p" event={"ID":"0fd710cc-405e-4c11-82b9-82e3d8ec793b","Type":"ContainerDied","Data":"11ecb13b78b5c19a0e4d1dc5c08932677321b6b27cfa9c2f9b7d294ab9d347b9"} Oct 30 21:12:44.345086 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:44.344951 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11ecb13b78b5c19a0e4d1dc5c08932677321b6b27cfa9c2f9b7d294ab9d347b9" Oct 30 21:12:45.399095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.399062 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh"] Oct 30 21:12:45.399613 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.399597 4926 cpu_manager.go:414] "containerMap: removing container" podUID="0fd710cc-405e-4c11-82b9-82e3d8ec793b" containerName="revoke-keys" Oct 30 21:12:45.399669 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.399616 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd710cc-405e-4c11-82b9-82e3d8ec793b" containerName="revoke-keys" Oct 30 21:12:45.399732 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.399720 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd710cc-405e-4c11-82b9-82e3d8ec793b" containerName="revoke-keys" Oct 30 21:12:45.402093 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.402075 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.416830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.416805 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-43c070-openshift-default-dockercfg-srjfd\"" Oct 30 21:12:45.432366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.432341 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh"] Oct 30 21:12:45.462581 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.462556 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-podinfo\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.462695 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.462605 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-std2t\" (UniqueName: \"kubernetes.io/projected/df81fda1-3052-4fb9-b9aa-5674a2e5847a-kube-api-access-std2t\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.462695 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.462627 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-envoy\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.462799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.462692 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-token\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.462924 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.462903 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-workload-certs\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.462994 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.462945 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-workload-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.462994 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.462977 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-data\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.463091 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.463073 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istiod-ca-cert\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.463139 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.463097 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-credential-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.564399 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.564370 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-workload-certs\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.564399 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.564405 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-workload-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.564598 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.564425 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-data\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.564633 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.564615 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istiod-ca-cert\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.564670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.564655 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-credential-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.564747 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.564723 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-podinfo\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.564820 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.564806 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-std2t\" (UniqueName: \"kubernetes.io/projected/df81fda1-3052-4fb9-b9aa-5674a2e5847a-kube-api-access-std2t\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.564873 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.564846 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-envoy\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.564921 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.564874 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-token\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.564921 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.564878 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-workload-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.564921 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.564908 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-workload-certs\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.565093 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.564959 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-data\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.565093 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.565002 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-credential-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.565518 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.565495 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istiod-ca-cert\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.567790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.567666 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-envoy\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.571370 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.571350 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2"] Oct 30 21:12:45.573770 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.573745 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-podinfo\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.575584 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.574729 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.590754 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.590729 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-token\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.590989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.590970 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-std2t\" (UniqueName: \"kubernetes.io/projected/df81fda1-3052-4fb9-b9aa-5674a2e5847a-kube-api-access-std2t\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.594486 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.593479 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-79477e-openshift-default-dockercfg-448fh\"" Oct 30 21:12:45.598407 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.598383 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2"] Oct 30 21:12:45.666482 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.666443 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-podinfo\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.666650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.666540 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-workload-certs\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.666650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.666589 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mvf5\" (UniqueName: \"kubernetes.io/projected/94ddad94-5958-43f7-b1fc-4c66a3117a29-kube-api-access-2mvf5\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.666762 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.666686 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ddad94-5958-43f7-b1fc-4c66a3117a29-istiod-ca-cert\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.666762 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.666728 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-workload-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.666864 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.666766 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-envoy\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.667012 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.666974 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-data\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.667151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.667057 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-token\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.667151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.667106 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-credential-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.717947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.717917 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:45.771589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.768654 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-podinfo\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.771589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.768695 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-workload-certs\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.771589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.768718 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mvf5\" (UniqueName: \"kubernetes.io/projected/94ddad94-5958-43f7-b1fc-4c66a3117a29-kube-api-access-2mvf5\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.771589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.768753 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ddad94-5958-43f7-b1fc-4c66a3117a29-istiod-ca-cert\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.771589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.768770 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-workload-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.771589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.768792 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-envoy\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.771589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.768816 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-data\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.771589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.768830 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-token\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.771589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.768858 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-credential-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.771589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.769294 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-credential-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.772957 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.772901 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ddad94-5958-43f7-b1fc-4c66a3117a29-istiod-ca-cert\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.773187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.773107 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-data\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.773284 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.773184 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-workload-certs\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.773446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.773409 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-workload-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.775755 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.775733 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-envoy\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.781537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.781173 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-podinfo\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.796686 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.796662 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-token\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.799333 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.799309 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mvf5\" (UniqueName: \"kubernetes.io/projected/94ddad94-5958-43f7-b1fc-4c66a3117a29-kube-api-access-2mvf5\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:45.926123 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:45.926054 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:46.061758 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.061733 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh"] Oct 30 21:12:46.063848 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:12:46.063822 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf81fda1_3052_4fb9_b9aa_5674a2e5847a.slice/crio-6cf8442391fceed5352c3a2915dc3c3c329e8e623d353480003229c21d293861 WatchSource:0}: Error finding container 6cf8442391fceed5352c3a2915dc3c3c329e8e623d353480003229c21d293861: Status 404 returned error can't find the container with id 6cf8442391fceed5352c3a2915dc3c3c329e8e623d353480003229c21d293861 Oct 30 21:12:46.066186 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.066154 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:12:46.066287 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.066228 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:12:46.066287 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.066275 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:12:46.179102 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.175232 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-842e1acf-mcm4q"] Oct 30 21:12:46.183055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.179892 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842e1acf-mcm4q" Oct 30 21:12:46.201052 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.198832 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:12:46.201052 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.198893 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:12:46.219593 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.219492 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-842e1acf-mcm4q"] Oct 30 21:12:46.238833 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.238809 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2"] Oct 30 21:12:46.241978 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.241933 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:12:46.242078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.242026 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:12:46.242132 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.242092 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:12:46.274553 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.274521 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a39e44-a5b5-46be-a81a-4d9b8c840037-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-842e1acf-mcm4q\" (UID: \"f5a39e44-a5b5-46be-a81a-4d9b8c840037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842e1acf-mcm4q" Oct 30 21:12:46.274655 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.274575 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a39e44-a5b5-46be-a81a-4d9b8c840037-maas-api-xks-ca\") pod \"maas-api-revoke-sub-842e1acf-mcm4q\" (UID: \"f5a39e44-a5b5-46be-a81a-4d9b8c840037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842e1acf-mcm4q" Oct 30 21:12:46.274695 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.274684 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8zqb\" (UniqueName: \"kubernetes.io/projected/f5a39e44-a5b5-46be-a81a-4d9b8c840037-kube-api-access-n8zqb\") pod \"maas-api-revoke-sub-842e1acf-mcm4q\" (UID: \"f5a39e44-a5b5-46be-a81a-4d9b8c840037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842e1acf-mcm4q" Oct 30 21:12:46.360305 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.360258 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" event={"ID":"df81fda1-3052-4fb9-b9aa-5674a2e5847a","Type":"ContainerStarted","Data":"097071ca8e288930d0fa88637ea631ebd5670fffe267c39222875736edc65c33"} Oct 30 21:12:46.360609 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.360586 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" event={"ID":"df81fda1-3052-4fb9-b9aa-5674a2e5847a","Type":"ContainerStarted","Data":"6cf8442391fceed5352c3a2915dc3c3c329e8e623d353480003229c21d293861"} Oct 30 21:12:46.362407 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.362381 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" event={"ID":"94ddad94-5958-43f7-b1fc-4c66a3117a29","Type":"ContainerStarted","Data":"a98ca3aa4acb654926715aafa87caffc7d3e4092ede54a439fa78e509d778072"} Oct 30 21:12:46.362647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.362617 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" event={"ID":"94ddad94-5958-43f7-b1fc-4c66a3117a29","Type":"ContainerStarted","Data":"946ccd877e9c9c4412aa216273577dc781cd9f000bdd30e795fd30c8f7fa841d"} Oct 30 21:12:46.375233 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.375206 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a39e44-a5b5-46be-a81a-4d9b8c840037-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-842e1acf-mcm4q\" (UID: \"f5a39e44-a5b5-46be-a81a-4d9b8c840037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842e1acf-mcm4q" Oct 30 21:12:46.375348 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.375250 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a39e44-a5b5-46be-a81a-4d9b8c840037-maas-api-xks-ca\") pod \"maas-api-revoke-sub-842e1acf-mcm4q\" (UID: \"f5a39e44-a5b5-46be-a81a-4d9b8c840037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842e1acf-mcm4q" Oct 30 21:12:46.375412 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.375363 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8zqb\" (UniqueName: \"kubernetes.io/projected/f5a39e44-a5b5-46be-a81a-4d9b8c840037-kube-api-access-n8zqb\") pod \"maas-api-revoke-sub-842e1acf-mcm4q\" (UID: \"f5a39e44-a5b5-46be-a81a-4d9b8c840037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842e1acf-mcm4q" Oct 30 21:12:46.376596 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.376562 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a39e44-a5b5-46be-a81a-4d9b8c840037-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-842e1acf-mcm4q\" (UID: \"f5a39e44-a5b5-46be-a81a-4d9b8c840037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842e1acf-mcm4q" Oct 30 21:12:46.387397 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.387375 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a39e44-a5b5-46be-a81a-4d9b8c840037-maas-api-xks-ca\") pod \"maas-api-revoke-sub-842e1acf-mcm4q\" (UID: \"f5a39e44-a5b5-46be-a81a-4d9b8c840037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842e1acf-mcm4q" Oct 30 21:12:46.395098 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.394259 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8zqb\" (UniqueName: \"kubernetes.io/projected/f5a39e44-a5b5-46be-a81a-4d9b8c840037-kube-api-access-n8zqb\") pod \"maas-api-revoke-sub-842e1acf-mcm4q\" (UID: \"f5a39e44-a5b5-46be-a81a-4d9b8c840037\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-842e1acf-mcm4q" Oct 30 21:12:46.409007 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.408949 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" podStartSLOduration=1.4089318450000001 podStartE2EDuration="1.408931845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:46.407446688 +0000 UTC m=+753.528520435" watchObservedRunningTime="2026-10-30 21:12:46.408931845 +0000 UTC m=+753.530005593" Oct 30 21:12:46.454256 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.454167 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" podStartSLOduration=1.454152258 podStartE2EDuration="1.454152258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:46.451798237 +0000 UTC m=+753.572871984" watchObservedRunningTime="2026-10-30 21:12:46.454152258 +0000 UTC m=+753.575226075" Oct 30 21:12:46.507084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.507019 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842e1acf-mcm4q" Oct 30 21:12:46.718555 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.718394 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:46.844276 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.844146 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-842e1acf-mcm4q"] Oct 30 21:12:46.862009 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:12:46.861975 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a39e44_a5b5_46be_a81a_4d9b8c840037.slice/crio-0c66c4a648636b2e5198fb4bc8db50b698af1cd243080c50d2f1a73e83e8f92f WatchSource:0}: Error finding container 0c66c4a648636b2e5198fb4bc8db50b698af1cd243080c50d2f1a73e83e8f92f: Status 404 returned error can't find the container with id 0c66c4a648636b2e5198fb4bc8db50b698af1cd243080c50d2f1a73e83e8f92f Oct 30 21:12:46.934104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:46.932566 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:47.395513 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:47.395471 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842e1acf-mcm4q" event={"ID":"f5a39e44-a5b5-46be-a81a-4d9b8c840037","Type":"ContainerStarted","Data":"0c66c4a648636b2e5198fb4bc8db50b698af1cd243080c50d2f1a73e83e8f92f"} Oct 30 21:12:47.743080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:47.731151 4926 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:12:47.743080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:47.731211 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" podUID="df81fda1-3052-4fb9-b9aa-5674a2e5847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 21:12:47.928336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:47.928299 4926 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:12:47.928516 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:47.928372 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" podUID="94ddad94-5958-43f7-b1fc-4c66a3117a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 21:12:48.131015 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.130963 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw"] Oct 30 21:12:48.134699 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.134682 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.163713 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.163548 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw"] Oct 30 21:12:48.247403 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.247375 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/217454ad-7acd-4b25-b91f-885bc464b6c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.247569 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.247420 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.247569 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.247506 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.247569 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.247550 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.247705 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.247616 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9njz\" (UniqueName: \"kubernetes.io/projected/217454ad-7acd-4b25-b91f-885bc464b6c5-kube-api-access-t9njz\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.247705 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.247688 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.247816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.247736 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.247816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.247793 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.247892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.247876 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.349715 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.349597 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9njz\" (UniqueName: \"kubernetes.io/projected/217454ad-7acd-4b25-b91f-885bc464b6c5-kube-api-access-t9njz\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.349715 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.349651 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.349715 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.349686 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.350060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.349734 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.350060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.349787 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.350060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.349854 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/217454ad-7acd-4b25-b91f-885bc464b6c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.350060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.349883 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.350060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.349918 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.350060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.349956 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.350994 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.350938 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.351127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.351004 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.351332 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.351311 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.351473 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.351366 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/217454ad-7acd-4b25-b91f-885bc464b6c5-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.351672 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.351652 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.353666 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.353644 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.355696 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.355674 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.430427 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.430391 4926 generic.go:356] "Generic (PLEG): container finished" podID="f5a39e44-a5b5-46be-a81a-4d9b8c840037" containerID="5f9ce042e4939ff63cd6725a4f5e6991e94b2335553a764aa33cb4b019b0e72a" exitCode=0 Oct 30 21:12:48.431053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.430940 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842e1acf-mcm4q" event={"ID":"f5a39e44-a5b5-46be-a81a-4d9b8c840037","Type":"ContainerDied","Data":"5f9ce042e4939ff63cd6725a4f5e6991e94b2335553a764aa33cb4b019b0e72a"} Oct 30 21:12:48.444234 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.444205 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.444401 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.444299 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9njz\" (UniqueName: \"kubernetes.io/projected/217454ad-7acd-4b25-b91f-885bc464b6c5-kube-api-access-t9njz\") pod \"maas-default-gateway-openshift-default-57f78bcd55-2tsvw\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.533280 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.533246 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:48.745584 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.745476 4926 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:12:48.745584 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.745558 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" podUID="df81fda1-3052-4fb9-b9aa-5674a2e5847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 21:12:48.943080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.942160 4926 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:12:48.943080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.942223 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" podUID="94ddad94-5958-43f7-b1fc-4c66a3117a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 21:12:48.943080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.943018 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw"] Oct 30 21:12:48.946674 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.946382 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:12:48.946674 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.946454 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:12:48.946674 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:48.946495 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:12:49.451053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:49.449652 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" event={"ID":"217454ad-7acd-4b25-b91f-885bc464b6c5","Type":"ContainerStarted","Data":"301f5a8ab1e97447872c54d0c6049409b4330f445905eb6671ca868a5fec20ad"} Oct 30 21:12:49.451053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:49.449732 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" event={"ID":"217454ad-7acd-4b25-b91f-885bc464b6c5","Type":"ContainerStarted","Data":"e3dd420f98e127cfc3996011e6d8a40644b96a7033470e47d310bf36a6ce75c8"} Oct 30 21:12:49.536062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:49.535142 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:49.776159 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:49.763227 4926 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:12:49.776159 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:49.763286 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" podUID="df81fda1-3052-4fb9-b9aa-5674a2e5847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 21:12:49.959360 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:49.952653 4926 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:12:49.959360 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:49.952725 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" podUID="94ddad94-5958-43f7-b1fc-4c66a3117a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 21:12:50.164913 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:50.162834 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842e1acf-mcm4q" Oct 30 21:12:50.191057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:50.190999 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5a39e44-a5b5-46be-a81a-4d9b8c840037-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a39e44-a5b5-46be-a81a-4d9b8c840037-maas-api-openshift-ca\") pod \"f5a39e44-a5b5-46be-a81a-4d9b8c840037\" (UID: \"f5a39e44-a5b5-46be-a81a-4d9b8c840037\") " Oct 30 21:12:50.191252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:50.191129 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5a39e44-a5b5-46be-a81a-4d9b8c840037-kube-api-access-n8zqb\" (UniqueName: \"kubernetes.io/projected/f5a39e44-a5b5-46be-a81a-4d9b8c840037-kube-api-access-n8zqb\") pod \"f5a39e44-a5b5-46be-a81a-4d9b8c840037\" (UID: \"f5a39e44-a5b5-46be-a81a-4d9b8c840037\") " Oct 30 21:12:50.191252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:50.191205 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5a39e44-a5b5-46be-a81a-4d9b8c840037-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a39e44-a5b5-46be-a81a-4d9b8c840037-maas-api-xks-ca\") pod \"f5a39e44-a5b5-46be-a81a-4d9b8c840037\" (UID: \"f5a39e44-a5b5-46be-a81a-4d9b8c840037\") " Oct 30 21:12:50.225262 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:50.215270 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5a39e44-a5b5-46be-a81a-4d9b8c840037-maas-api-openshift-ca" pod "f5a39e44-a5b5-46be-a81a-4d9b8c840037" (UID: "f5a39e44-a5b5-46be-a81a-4d9b8c840037"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:12:50.242261 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:50.227511 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a39e44-a5b5-46be-a81a-4d9b8c840037-kube-api-access-n8zqb" pod "f5a39e44-a5b5-46be-a81a-4d9b8c840037" (UID: "f5a39e44-a5b5-46be-a81a-4d9b8c840037"). InnerVolumeSpecName "kube-api-access-n8zqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:12:50.242261 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:50.228102 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5a39e44-a5b5-46be-a81a-4d9b8c840037-maas-api-xks-ca" pod "f5a39e44-a5b5-46be-a81a-4d9b8c840037" (UID: "f5a39e44-a5b5-46be-a81a-4d9b8c840037"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:12:50.242261 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:50.229080 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" podStartSLOduration=2.22906223 podStartE2EDuration="2.22906223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:12:49.500364875 +0000 UTC m=+756.621438622" watchObservedRunningTime="2026-10-30 21:12:50.22906223 +0000 UTC m=+757.350135977" Oct 30 21:12:50.295924 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:50.295829 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a39e44-a5b5-46be-a81a-4d9b8c840037-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:12:50.296164 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:50.296143 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8zqb\" (UniqueName: \"kubernetes.io/projected/f5a39e44-a5b5-46be-a81a-4d9b8c840037-kube-api-access-n8zqb\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:12:50.296286 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:50.296272 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a39e44-a5b5-46be-a81a-4d9b8c840037-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:12:50.544057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:50.540321 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842e1acf-mcm4q" Oct 30 21:12:50.544057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:50.542094 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-842e1acf-mcm4q" event={"ID":"f5a39e44-a5b5-46be-a81a-4d9b8c840037","Type":"ContainerDied","Data":"0c66c4a648636b2e5198fb4bc8db50b698af1cd243080c50d2f1a73e83e8f92f"} Oct 30 21:12:50.544057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:50.542149 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c66c4a648636b2e5198fb4bc8db50b698af1cd243080c50d2f1a73e83e8f92f" Oct 30 21:12:50.552282 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:50.552204 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-2tsvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:12:50.552531 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:50.552501 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" podUID="217454ad-7acd-4b25-b91f-885bc464b6c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 21:12:50.727698 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:50.727648 4926 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:12:50.757386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:50.757333 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" podUID="df81fda1-3052-4fb9-b9aa-5674a2e5847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 21:12:50.939057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:50.927391 4926 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:12:50.939057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:50.927450 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" podUID="94ddad94-5958-43f7-b1fc-4c66a3117a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 21:12:51.091069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:51.090442 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:51.503058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:51.492151 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 21:12:51.525514 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:51.525465 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:12:51.535609 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:51.535578 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-2tsvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:12:51.535864 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:51.535829 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" podUID="217454ad-7acd-4b25-b91f-885bc464b6c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 21:12:51.568415 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:51.568377 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:12:51.633365 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:51.633192 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:12:51.673445 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:51.673403 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:12:51.724064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:51.723517 4926 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:12:51.724064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:51.723591 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" podUID="df81fda1-3052-4fb9-b9aa-5674a2e5847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 21:12:51.755634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:51.755545 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:51.807186 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:51.799151 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:12:51.851490 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:51.851436 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:12:51.874479 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:51.874435 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:12:51.931495 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:51.931456 4926 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:12:51.931777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:51.931743 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" podUID="94ddad94-5958-43f7-b1fc-4c66a3117a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 21:12:52.058951 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:52.058848 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:12:52.278057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:52.277556 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:12:52.497057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:52.495166 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:12:52.497057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:52.495802 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:12:52.530055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:52.529126 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:12:52.530055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:52.529491 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 21:12:52.530055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:52.529880 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:12:52.577265 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:52.577212 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:12:52.577979 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:52.577956 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:12:52.599082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:52.585283 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:12:52.651056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:52.649154 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-2tsvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:12:52.651056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:52.649215 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" podUID="217454ad-7acd-4b25-b91f-885bc464b6c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 21:12:52.729389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:52.661723 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:12:52.729389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:52.702223 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:12:52.732657 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:52.732624 4926 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:12:52.732886 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:52.732858 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" podUID="df81fda1-3052-4fb9-b9aa-5674a2e5847a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 21:12:52.741213 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:52.740820 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:12:52.926771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:52.926725 4926 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:12:52.926952 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:52.926785 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" podUID="94ddad94-5958-43f7-b1fc-4c66a3117a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 21:12:53.534432 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:53.534391 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-2tsvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:12:53.535054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:53.535002 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" podUID="217454ad-7acd-4b25-b91f-885bc464b6c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 21:12:54.047063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:54.046993 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:54.047624 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:54.047598 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:54.048652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:54.048605 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:12:54.536751 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:54.536057 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-2tsvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:12:54.536751 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:54.536106 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" podUID="217454ad-7acd-4b25-b91f-885bc464b6c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 21:12:54.970211 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:54.970160 4926 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:12:54.970504 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:54.970456 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" podUID="94ddad94-5958-43f7-b1fc-4c66a3117a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:12:55.231339 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:55.231242 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:55.232642 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:55.231873 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:55.233127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:55.233108 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:12:55.534328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:55.534248 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-2tsvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:12:55.534328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:55.534303 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" podUID="217454ad-7acd-4b25-b91f-885bc464b6c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 21:12:56.534831 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:56.534791 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-2tsvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:12:56.535330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:56.534845 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" podUID="217454ad-7acd-4b25-b91f-885bc464b6c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 21:12:57.534806 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:57.534764 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-2tsvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:12:57.535133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:57.534835 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" podUID="217454ad-7acd-4b25-b91f-885bc464b6c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 30 21:12:58.940509 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:58.940478 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:58.941026 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:58.940903 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:12:58.941857 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:12:58.941836 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:13:00.238173 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.238135 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw"] Oct 30 21:13:00.238770 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.238743 4926 cpu_manager.go:414] "containerMap: removing container" podUID="f5a39e44-a5b5-46be-a81a-4d9b8c840037" containerName="revoke-keys" Oct 30 21:13:00.238770 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.238766 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5a39e44-a5b5-46be-a81a-4d9b8c840037" containerName="revoke-keys" Oct 30 21:13:00.238986 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.238929 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5a39e44-a5b5-46be-a81a-4d9b8c840037" containerName="revoke-keys" Oct 30 21:13:00.242279 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.242262 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.256167 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.256147 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-58fea9-openshift-default-dockercfg-2wlz5\"" Oct 30 21:13:00.271986 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.271958 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw"] Oct 30 21:13:00.339834 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.339802 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-token\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.339834 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.339835 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxvv7\" (UniqueName: \"kubernetes.io/projected/2faae7ee-cee5-49cc-a4e2-22adc43c8289-kube-api-access-mxvv7\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.340011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.339859 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istiod-ca-cert\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.340011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.339936 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-data\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.340110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.340008 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-podinfo\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.340145 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.340123 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-workload-certs\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.340183 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.340167 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-credential-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.340216 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.340198 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-envoy\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.340251 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.340227 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-workload-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.441485 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.441451 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-token\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.441660 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.441493 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxvv7\" (UniqueName: \"kubernetes.io/projected/2faae7ee-cee5-49cc-a4e2-22adc43c8289-kube-api-access-mxvv7\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.441660 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.441522 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istiod-ca-cert\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.441660 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.441542 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-data\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.441660 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.441600 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-podinfo\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.441844 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.441670 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-workload-certs\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.441844 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.441723 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-credential-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.441844 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.441768 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-envoy\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.441991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.441907 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-workload-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.442116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.442093 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-data\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.442191 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.442130 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-workload-certs\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.442315 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.442292 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-workload-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.442381 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.442343 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istiod-ca-cert\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.442381 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.442362 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-credential-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.444143 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.444119 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-envoy\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.444336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.444317 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-podinfo\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.460053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.459540 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxvv7\" (UniqueName: \"kubernetes.io/projected/2faae7ee-cee5-49cc-a4e2-22adc43c8289-kube-api-access-mxvv7\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.460053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.459949 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-token\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.555232 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.555156 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:00.629467 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.629430 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg"] Oct 30 21:13:00.636592 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.635942 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.645670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.645013 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-data\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.645670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.645083 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-podinfo\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.645670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.645177 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6mkm\" (UniqueName: \"kubernetes.io/projected/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-kube-api-access-w6mkm\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.645670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.645213 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-workload-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.645670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.645243 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-token\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.645670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.645307 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-workload-certs\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.645670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.645388 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-credential-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.645670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.645435 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-envoy\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.645670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.645464 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istiod-ca-cert\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.656921 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.656901 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-2cda44-openshift-default-dockercfg-4kbl9\"" Oct 30 21:13:00.665108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.665081 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg"] Oct 30 21:13:00.746222 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.746185 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-workload-certs\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.746420 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.746250 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-credential-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.746420 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.746297 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-envoy\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.746420 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.746322 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istiod-ca-cert\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.746420 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.746366 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-data\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.746420 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.746392 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-podinfo\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.746702 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.746469 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6mkm\" (UniqueName: \"kubernetes.io/projected/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-kube-api-access-w6mkm\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.746702 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.746506 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-workload-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.746702 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.746538 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-token\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.746702 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.746662 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-workload-certs\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.746939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.746897 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-data\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.747124 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.747103 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-credential-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.747246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.747173 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istiod-ca-cert\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.747246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.747192 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-workload-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.749198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.749180 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-podinfo\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.749265 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.749231 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-envoy\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.766814 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.766784 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6mkm\" (UniqueName: \"kubernetes.io/projected/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-kube-api-access-w6mkm\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.768079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.768055 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-token\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:00.882928 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.882900 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw"] Oct 30 21:13:00.886194 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:13:00.885632 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2faae7ee_cee5_49cc_a4e2_22adc43c8289.slice/crio-0480e2dc8d8f324599ec615de1910c45dda58440e8a20f01f36d5bfbb27a6c8e WatchSource:0}: Error finding container 0480e2dc8d8f324599ec615de1910c45dda58440e8a20f01f36d5bfbb27a6c8e: Status 404 returned error can't find the container with id 0480e2dc8d8f324599ec615de1910c45dda58440e8a20f01f36d5bfbb27a6c8e Oct 30 21:13:00.888248 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.888214 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:00.888367 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.888289 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:00.888367 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.888335 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:00.958741 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:00.958717 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:01.304857 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:01.304826 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg"] Oct 30 21:13:01.305268 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:13:01.304837 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71625f0_cf7b_4c3e_8e7c_fe0c6d8d9d70.slice/crio-574e1565b274d545439a79d559532448a46a1e6fee52384eb45ac4eedf467cf2 WatchSource:0}: Error finding container 574e1565b274d545439a79d559532448a46a1e6fee52384eb45ac4eedf467cf2: Status 404 returned error can't find the container with id 574e1565b274d545439a79d559532448a46a1e6fee52384eb45ac4eedf467cf2 Oct 30 21:13:01.307905 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:01.307871 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:01.308064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:01.307937 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:01.308064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:01.307962 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:01.848541 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:01.848502 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" event={"ID":"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70","Type":"ContainerStarted","Data":"8ac910cc08ef3b7b28ecadebbbd09f8927f9505427669f6e06028fb949ea4069"} Oct 30 21:13:01.848758 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:01.848565 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" event={"ID":"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70","Type":"ContainerStarted","Data":"574e1565b274d545439a79d559532448a46a1e6fee52384eb45ac4eedf467cf2"} Oct 30 21:13:01.851250 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:01.851220 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" event={"ID":"2faae7ee-cee5-49cc-a4e2-22adc43c8289","Type":"ContainerStarted","Data":"e88570d94185fb9bf8d1edb61636c843f30baa828e2eaad1976e05d1c555f0fc"} Oct 30 21:13:01.851367 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:01.851275 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" event={"ID":"2faae7ee-cee5-49cc-a4e2-22adc43c8289","Type":"ContainerStarted","Data":"0480e2dc8d8f324599ec615de1910c45dda58440e8a20f01f36d5bfbb27a6c8e"} Oct 30 21:13:01.918070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:01.917956 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" podStartSLOduration=1.917937868 podStartE2EDuration="1.917937868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:01.902477994 +0000 UTC m=+769.023551742" watchObservedRunningTime="2026-10-30 21:13:01.917937868 +0000 UTC m=+769.039011616" Oct 30 21:13:01.946004 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:01.945945 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" podStartSLOduration=1.945927024 podStartE2EDuration="1.945927024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:01.945178686 +0000 UTC m=+769.066252432" watchObservedRunningTime="2026-10-30 21:13:01.945927024 +0000 UTC m=+769.067000772" Oct 30 21:13:01.959381 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:01.959355 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:02.556246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:02.556211 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:02.557692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:02.557671 4926 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:13:02.557759 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:02.557712 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" podUID="2faae7ee-cee5-49cc-a4e2-22adc43c8289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 21:13:02.960214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:02.960175 4926 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:13:02.960416 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:02.960232 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" podUID="b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:13:03.480056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:03.478684 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a3cb196-mb2f2"] Oct 30 21:13:03.495343 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:03.495315 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a3cb196-mb2f2"] Oct 30 21:13:03.495630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:03.495615 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3cb196-mb2f2" Oct 30 21:13:03.519055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:03.515865 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:13:03.543058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:03.536792 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:13:03.556069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:03.556025 4926 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:13:03.556338 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:03.556307 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" podUID="2faae7ee-cee5-49cc-a4e2-22adc43c8289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 21:13:03.638056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:03.629410 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw7xl\" (UniqueName: \"kubernetes.io/projected/1a329ae7-103e-483a-9e33-94276f6f4d72-kube-api-access-mw7xl\") pod \"maas-api-revoke-sub-3a3cb196-mb2f2\" (UID: \"1a329ae7-103e-483a-9e33-94276f6f4d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3cb196-mb2f2" Oct 30 21:13:03.638056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:03.629580 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a329ae7-103e-483a-9e33-94276f6f4d72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a3cb196-mb2f2\" (UID: \"1a329ae7-103e-483a-9e33-94276f6f4d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3cb196-mb2f2" Oct 30 21:13:03.638056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:03.629642 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a329ae7-103e-483a-9e33-94276f6f4d72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a3cb196-mb2f2\" (UID: \"1a329ae7-103e-483a-9e33-94276f6f4d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3cb196-mb2f2" Oct 30 21:13:03.743899 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:03.737663 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw7xl\" (UniqueName: \"kubernetes.io/projected/1a329ae7-103e-483a-9e33-94276f6f4d72-kube-api-access-mw7xl\") pod \"maas-api-revoke-sub-3a3cb196-mb2f2\" (UID: \"1a329ae7-103e-483a-9e33-94276f6f4d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3cb196-mb2f2" Oct 30 21:13:03.743899 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:03.737798 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a329ae7-103e-483a-9e33-94276f6f4d72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a3cb196-mb2f2\" (UID: \"1a329ae7-103e-483a-9e33-94276f6f4d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3cb196-mb2f2" Oct 30 21:13:03.743899 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:03.737850 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a329ae7-103e-483a-9e33-94276f6f4d72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a3cb196-mb2f2\" (UID: \"1a329ae7-103e-483a-9e33-94276f6f4d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3cb196-mb2f2" Oct 30 21:13:03.743899 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:03.738664 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a329ae7-103e-483a-9e33-94276f6f4d72-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a3cb196-mb2f2\" (UID: \"1a329ae7-103e-483a-9e33-94276f6f4d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3cb196-mb2f2" Oct 30 21:13:03.744983 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:03.744958 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a329ae7-103e-483a-9e33-94276f6f4d72-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a3cb196-mb2f2\" (UID: \"1a329ae7-103e-483a-9e33-94276f6f4d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3cb196-mb2f2" Oct 30 21:13:03.759393 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:03.759369 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw7xl\" (UniqueName: \"kubernetes.io/projected/1a329ae7-103e-483a-9e33-94276f6f4d72-kube-api-access-mw7xl\") pod \"maas-api-revoke-sub-3a3cb196-mb2f2\" (UID: \"1a329ae7-103e-483a-9e33-94276f6f4d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3cb196-mb2f2" Oct 30 21:13:03.863045 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:03.863011 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3cb196-mb2f2" Oct 30 21:13:03.959276 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:03.959243 4926 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:13:03.959494 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:03.959473 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" podUID="b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:13:04.363754 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:04.363717 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a3cb196-mb2f2"] Oct 30 21:13:04.557461 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:04.556178 4926 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:13:04.557461 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:04.556236 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" podUID="2faae7ee-cee5-49cc-a4e2-22adc43c8289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 21:13:04.882080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:04.879057 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3cb196-mb2f2" event={"ID":"1a329ae7-103e-483a-9e33-94276f6f4d72","Type":"ContainerStarted","Data":"80ce75149c1f993ba32731c51e2f74d470bab6cdb9edb61628b1f3d172292348"} Oct 30 21:13:04.959626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:04.959590 4926 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:13:04.959782 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:04.959647 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" podUID="b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:13:05.047893 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:05.047859 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:13:05.557063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:05.555702 4926 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:13:05.557063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:05.555759 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" podUID="2faae7ee-cee5-49cc-a4e2-22adc43c8289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 21:13:05.899472 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:05.899437 4926 generic.go:356] "Generic (PLEG): container finished" podID="1a329ae7-103e-483a-9e33-94276f6f4d72" containerID="4c96d6646d242fc8b6e397b29948355d6161ceacf05170033b096a5e159d703e" exitCode=0 Oct 30 21:13:05.900201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:05.900174 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3cb196-mb2f2" event={"ID":"1a329ae7-103e-483a-9e33-94276f6f4d72","Type":"ContainerDied","Data":"4c96d6646d242fc8b6e397b29948355d6161ceacf05170033b096a5e159d703e"} Oct 30 21:13:05.962471 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:05.962395 4926 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:13:05.962692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:05.962671 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" podUID="b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:13:06.555567 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:06.555523 4926 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:13:06.555781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:06.555593 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" podUID="2faae7ee-cee5-49cc-a4e2-22adc43c8289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 21:13:06.981629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:06.981174 4926 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:13:06.981629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:06.981252 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" podUID="b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:13:07.364753 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:07.364727 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3cb196-mb2f2" Oct 30 21:13:07.445052 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:07.445009 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a329ae7-103e-483a-9e33-94276f6f4d72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a329ae7-103e-483a-9e33-94276f6f4d72-maas-api-openshift-ca\") pod \"1a329ae7-103e-483a-9e33-94276f6f4d72\" (UID: \"1a329ae7-103e-483a-9e33-94276f6f4d72\") " Oct 30 21:13:07.445393 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:07.445379 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a329ae7-103e-483a-9e33-94276f6f4d72-kube-api-access-mw7xl\" (UniqueName: \"kubernetes.io/projected/1a329ae7-103e-483a-9e33-94276f6f4d72-kube-api-access-mw7xl\") pod \"1a329ae7-103e-483a-9e33-94276f6f4d72\" (UID: \"1a329ae7-103e-483a-9e33-94276f6f4d72\") " Oct 30 21:13:07.445527 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:07.445516 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a329ae7-103e-483a-9e33-94276f6f4d72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a329ae7-103e-483a-9e33-94276f6f4d72-maas-api-xks-ca\") pod \"1a329ae7-103e-483a-9e33-94276f6f4d72\" (UID: \"1a329ae7-103e-483a-9e33-94276f6f4d72\") " Oct 30 21:13:07.446571 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:07.446546 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a329ae7-103e-483a-9e33-94276f6f4d72-maas-api-openshift-ca" pod "1a329ae7-103e-483a-9e33-94276f6f4d72" (UID: "1a329ae7-103e-483a-9e33-94276f6f4d72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:13:07.448553 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:07.448532 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a329ae7-103e-483a-9e33-94276f6f4d72-maas-api-xks-ca" pod "1a329ae7-103e-483a-9e33-94276f6f4d72" (UID: "1a329ae7-103e-483a-9e33-94276f6f4d72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:13:07.451853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:07.451819 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a329ae7-103e-483a-9e33-94276f6f4d72-kube-api-access-mw7xl" pod "1a329ae7-103e-483a-9e33-94276f6f4d72" (UID: "1a329ae7-103e-483a-9e33-94276f6f4d72"). InnerVolumeSpecName "kube-api-access-mw7xl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:13:07.530846 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:13:07.530813 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:13:07.546974 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:07.546907 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw7xl\" (UniqueName: \"kubernetes.io/projected/1a329ae7-103e-483a-9e33-94276f6f4d72-kube-api-access-mw7xl\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:13:07.546974 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:07.546933 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a329ae7-103e-483a-9e33-94276f6f4d72-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:13:07.546974 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:07.546943 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a329ae7-103e-483a-9e33-94276f6f4d72-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:13:07.555819 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:07.555795 4926 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 30 21:13:07.555925 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:07.555841 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" podUID="2faae7ee-cee5-49cc-a4e2-22adc43c8289" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 30 21:13:07.928580 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:07.928537 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3cb196-mb2f2" Oct 30 21:13:07.928580 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:07.928551 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a3cb196-mb2f2" event={"ID":"1a329ae7-103e-483a-9e33-94276f6f4d72","Type":"ContainerDied","Data":"80ce75149c1f993ba32731c51e2f74d470bab6cdb9edb61628b1f3d172292348"} Oct 30 21:13:07.928833 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:07.928631 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80ce75149c1f993ba32731c51e2f74d470bab6cdb9edb61628b1f3d172292348" Oct 30 21:13:07.961345 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:07.960211 4926 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:13:07.961345 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:07.960268 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" podUID="b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:13:08.560855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:08.560824 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:08.561302 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:08.561252 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:08.561940 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:08.561915 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:13:08.859156 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:08.859063 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb"] Oct 30 21:13:08.859812 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:08.859791 4926 cpu_manager.go:414] "containerMap: removing container" podUID="1a329ae7-103e-483a-9e33-94276f6f4d72" containerName="revoke-keys" Oct 30 21:13:08.859932 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:08.859812 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a329ae7-103e-483a-9e33-94276f6f4d72" containerName="revoke-keys" Oct 30 21:13:08.859992 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:08.859975 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a329ae7-103e-483a-9e33-94276f6f4d72" containerName="revoke-keys" Oct 30 21:13:08.863982 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:08.863958 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:08.884158 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:08.883665 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ed088d\"/\"distinct-w6-ed088d-kserve-self-signed-certs\"" Oct 30 21:13:08.964536 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:08.964498 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:08.964956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:08.964933 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:08.965714 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:08.965696 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:13:08.966220 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:08.966202 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ae4a221-e691-49b2-bafa-35e50ea1bead-tls-certs\") pod \"distinct-w6-ed088d-kserve-54f5f9f75d-424qb\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:08.966312 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:08.966236 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-home\") pod \"distinct-w6-ed088d-kserve-54f5f9f75d-424qb\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:08.966312 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:08.966295 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdgfx\" (UniqueName: \"kubernetes.io/projected/2ae4a221-e691-49b2-bafa-35e50ea1bead-kube-api-access-zdgfx\") pod \"distinct-w6-ed088d-kserve-54f5f9f75d-424qb\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:08.966449 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:08.966423 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-model-cache\") pod \"distinct-w6-ed088d-kserve-54f5f9f75d-424qb\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:08.966562 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:08.966474 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-tmp-dir\") pod \"distinct-w6-ed088d-kserve-54f5f9f75d-424qb\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:08.966638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:08.966590 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-dshm\") pod \"distinct-w6-ed088d-kserve-54f5f9f75d-424qb\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:08.977595 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:08.977569 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb"] Oct 30 21:13:09.067610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:09.067577 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-dshm\") pod \"distinct-w6-ed088d-kserve-54f5f9f75d-424qb\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:09.067821 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:09.067734 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ae4a221-e691-49b2-bafa-35e50ea1bead-tls-certs\") pod \"distinct-w6-ed088d-kserve-54f5f9f75d-424qb\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:09.067821 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:09.067764 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-home\") pod \"distinct-w6-ed088d-kserve-54f5f9f75d-424qb\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:09.067945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:09.067871 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdgfx\" (UniqueName: \"kubernetes.io/projected/2ae4a221-e691-49b2-bafa-35e50ea1bead-kube-api-access-zdgfx\") pod \"distinct-w6-ed088d-kserve-54f5f9f75d-424qb\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:09.067945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:09.067905 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-model-cache\") pod \"distinct-w6-ed088d-kserve-54f5f9f75d-424qb\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:09.068069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:09.067944 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-tmp-dir\") pod \"distinct-w6-ed088d-kserve-54f5f9f75d-424qb\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:09.068696 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:09.068378 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-home\") pod \"distinct-w6-ed088d-kserve-54f5f9f75d-424qb\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:09.069136 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:09.069108 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-tmp-dir\") pod \"distinct-w6-ed088d-kserve-54f5f9f75d-424qb\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:09.069314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:09.069281 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-model-cache\") pod \"distinct-w6-ed088d-kserve-54f5f9f75d-424qb\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:09.070866 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:09.070845 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ae4a221-e691-49b2-bafa-35e50ea1bead-tls-certs\") pod \"distinct-w6-ed088d-kserve-54f5f9f75d-424qb\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:09.071487 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:09.071459 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-dshm\") pod \"distinct-w6-ed088d-kserve-54f5f9f75d-424qb\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:09.118101 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:09.113516 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdgfx\" (UniqueName: \"kubernetes.io/projected/2ae4a221-e691-49b2-bafa-35e50ea1bead-kube-api-access-zdgfx\") pod \"distinct-w6-ed088d-kserve-54f5f9f75d-424qb\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:09.193072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:09.191672 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:09.504429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:09.504387 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb"] Oct 30 21:13:09.509519 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:13:09.509489 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ae4a221_e691_49b2_bafa_35e50ea1bead.slice/crio-00cf3d05415e37da2476d12c8e3e6db69eafd2277f5c4ca5c8a8e32c47e41b5b WatchSource:0}: Error finding container 00cf3d05415e37da2476d12c8e3e6db69eafd2277f5c4ca5c8a8e32c47e41b5b: Status 404 returned error can't find the container with id 00cf3d05415e37da2476d12c8e3e6db69eafd2277f5c4ca5c8a8e32c47e41b5b Oct 30 21:13:09.944490 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:09.944448 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" event={"ID":"2ae4a221-e691-49b2-bafa-35e50ea1bead","Type":"ContainerStarted","Data":"eb680a58d7702a24a9211fd9c44d41083d67581d1a06a11f418a084f92f00acf"} Oct 30 21:13:09.944965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:09.944501 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" event={"ID":"2ae4a221-e691-49b2-bafa-35e50ea1bead","Type":"ContainerStarted","Data":"00cf3d05415e37da2476d12c8e3e6db69eafd2277f5c4ca5c8a8e32c47e41b5b"} Oct 30 21:13:09.982811 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:09.982760 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podStartSLOduration=1.982746734 podStartE2EDuration="1.982746734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:09.981550896 +0000 UTC m=+777.102624642" watchObservedRunningTime="2026-10-30 21:13:09.982746734 +0000 UTC m=+777.103820479" Oct 30 21:13:10.192068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:10.192025 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:15.578761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.578719 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n"] Oct 30 21:13:15.584383 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.584361 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.616386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.615390 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n"] Oct 30 21:13:15.694207 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.694180 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-56ac74-openshift-default-dockercfg-gsffb\"" Oct 30 21:13:15.755414 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.755385 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-data\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.755562 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.755419 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94eead01-fd4c-4c47-8d70-527ba6643cd5-istiod-ca-cert\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.755600 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.755564 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-workload-certs\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.755638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.755602 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-podinfo\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.755730 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.755709 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-token\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.755768 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.755761 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-envoy\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.755853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.755839 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-workload-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.755892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.755862 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8jw5\" (UniqueName: \"kubernetes.io/projected/94eead01-fd4c-4c47-8d70-527ba6643cd5-kube-api-access-g8jw5\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.755944 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.755889 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-credential-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.856662 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.856555 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-workload-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.856662 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.856606 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8jw5\" (UniqueName: \"kubernetes.io/projected/94eead01-fd4c-4c47-8d70-527ba6643cd5-kube-api-access-g8jw5\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.856662 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.856644 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-credential-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.856967 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.856682 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-data\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.856967 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.856710 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94eead01-fd4c-4c47-8d70-527ba6643cd5-istiod-ca-cert\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.856967 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.856824 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-workload-certs\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.856967 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.856870 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-podinfo\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.857233 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.856968 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-token\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.857233 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.857023 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-workload-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.857233 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.857070 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-envoy\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.857749 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.857719 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-credential-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.857918 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.857890 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94eead01-fd4c-4c47-8d70-527ba6643cd5-istiod-ca-cert\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.858020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.857994 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-workload-certs\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.859158 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.858328 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-data\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.859865 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.859840 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-envoy\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.859961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.859913 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-podinfo\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.876096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.876069 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8jw5\" (UniqueName: \"kubernetes.io/projected/94eead01-fd4c-4c47-8d70-527ba6643cd5-kube-api-access-g8jw5\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.876254 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.876231 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-token\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:15.899484 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:15.899461 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:16.162464 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.162392 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm"] Oct 30 21:13:16.167495 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.167474 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.196252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.196219 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm"] Oct 30 21:13:16.206213 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.206185 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-ed088d-openshift-default-dockercfg-xqfc5\"" Oct 30 21:13:16.227676 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.227646 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n"] Oct 30 21:13:16.228180 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:13:16.228152 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94eead01_fd4c_4c47_8d70_527ba6643cd5.slice/crio-d4a1dc274a3033f63b354f5210b4c7dc44aa0ba72b5be99c97a5c5b8ce95294a WatchSource:0}: Error finding container d4a1dc274a3033f63b354f5210b4c7dc44aa0ba72b5be99c97a5c5b8ce95294a: Status 404 returned error can't find the container with id d4a1dc274a3033f63b354f5210b4c7dc44aa0ba72b5be99c97a5c5b8ce95294a Oct 30 21:13:16.231377 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.231335 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:16.231470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.231430 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:16.231507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.231482 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:16.260862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.260828 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-data\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.260862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.260865 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-workload-certs\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.261085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.260931 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-envoy\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.261085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.260954 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-workload-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.261085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.260974 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-podinfo\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.261085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.260999 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-token\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.261246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.261126 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-credential-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.261246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.261175 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istiod-ca-cert\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.261246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.261234 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92lhn\" (UniqueName: \"kubernetes.io/projected/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-kube-api-access-92lhn\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.362639 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.362599 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istiod-ca-cert\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.362811 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.362655 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92lhn\" (UniqueName: \"kubernetes.io/projected/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-kube-api-access-92lhn\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.362811 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.362787 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-data\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.362811 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.362810 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-workload-certs\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.362997 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.362839 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-envoy\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.362997 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.362868 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-workload-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.362997 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.362889 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-podinfo\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.362997 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.362914 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-token\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.362997 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.362944 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-credential-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.363399 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.363371 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-data\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.363476 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.363455 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istiod-ca-cert\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.363535 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.363498 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-workload-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.363645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.363626 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-workload-certs\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.363763 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.363735 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-credential-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.365767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.365740 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-envoy\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.365868 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.365774 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-podinfo\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.385157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.385128 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92lhn\" (UniqueName: \"kubernetes.io/projected/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-kube-api-access-92lhn\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.385296 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.385213 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-token\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.485629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.485598 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:16.819363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.819331 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm"] Oct 30 21:13:16.820570 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:13:16.820536 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc674d3b2_72ca_4b18_9e7c_c6dcdd0ee7b7.slice/crio-e54255a2196dea3a432a1a19bfc59a3766a2d18610b9e4c69050944b5347e64b WatchSource:0}: Error finding container e54255a2196dea3a432a1a19bfc59a3766a2d18610b9e4c69050944b5347e64b: Status 404 returned error can't find the container with id e54255a2196dea3a432a1a19bfc59a3766a2d18610b9e4c69050944b5347e64b Oct 30 21:13:16.823023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.822995 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:16.823089 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.823077 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:16.823131 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:16.823109 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:17.024062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:17.024009 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" event={"ID":"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7","Type":"ContainerStarted","Data":"c9ae83175007b668ac93f22fb889cad519f1241de85540d01101e5a57709ccff"} Oct 30 21:13:17.025060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:17.025012 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" event={"ID":"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7","Type":"ContainerStarted","Data":"e54255a2196dea3a432a1a19bfc59a3766a2d18610b9e4c69050944b5347e64b"} Oct 30 21:13:17.025127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:17.025075 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" event={"ID":"94eead01-fd4c-4c47-8d70-527ba6643cd5","Type":"ContainerStarted","Data":"fa203018dbac282594a360fb682cc11b3fc41f5488fb63f241ebec303b46bc95"} Oct 30 21:13:17.025127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:17.025117 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" event={"ID":"94eead01-fd4c-4c47-8d70-527ba6643cd5","Type":"ContainerStarted","Data":"d4a1dc274a3033f63b354f5210b4c7dc44aa0ba72b5be99c97a5c5b8ce95294a"} Oct 30 21:13:17.064616 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:17.064541 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" podStartSLOduration=1.064521893 podStartE2EDuration="1.064521893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:17.064308249 +0000 UTC m=+784.185381995" watchObservedRunningTime="2026-10-30 21:13:17.064521893 +0000 UTC m=+784.185595642" Oct 30 21:13:17.111642 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:17.111522 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" podStartSLOduration=2.111503745 podStartE2EDuration="2.111503745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:17.110877184 +0000 UTC m=+784.231950934" watchObservedRunningTime="2026-10-30 21:13:17.111503745 +0000 UTC m=+784.232577491" Oct 30 21:13:17.494373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:17.494338 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:17.508827 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:17.508488 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:13:17.508827 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:17.508552 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" podUID="c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:13:17.910023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:17.909938 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:17.910725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:17.910699 4926 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:13:17.914054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:17.910948 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" podUID="94eead01-fd4c-4c47-8d70-527ba6643cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 21:13:18.508975 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:18.508923 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:13:18.529351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:18.529277 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" podUID="c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:13:18.907112 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:18.906953 4926 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:13:18.907112 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:18.907021 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" podUID="94eead01-fd4c-4c47-8d70-527ba6643cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 21:13:19.195053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:19.194268 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:19.280055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:19.279272 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:19.492127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:19.489370 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:13:19.492127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:19.489429 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" podUID="c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:13:19.903112 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:19.902461 4926 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:13:19.903112 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:19.902533 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" podUID="94eead01-fd4c-4c47-8d70-527ba6643cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 21:13:20.455539 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:20.455328 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:13:20.543480 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:20.543449 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:13:20.543726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:20.543699 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" podUID="c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:13:20.902116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:20.900409 4926 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:13:20.902116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:20.900474 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" podUID="94eead01-fd4c-4c47-8d70-527ba6643cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 21:13:21.493606 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:21.493560 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:13:21.494102 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:21.493632 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" podUID="c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:13:21.909115 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:21.901495 4926 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:13:21.909115 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:21.901569 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" podUID="94eead01-fd4c-4c47-8d70-527ba6643cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 21:13:22.489450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:22.489405 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:13:22.489676 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:22.489474 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" podUID="c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:13:22.903394 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:22.903288 4926 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Oct 30 21:13:22.903394 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:22.903363 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" podUID="94eead01-fd4c-4c47-8d70-527ba6643cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Oct 30 21:13:23.486452 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:23.486405 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:13:23.486682 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:23.486477 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" podUID="c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:13:24.432915 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:24.432874 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:24.433914 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:24.433891 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:24.435064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:24.435006 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:13:25.170153 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:25.170104 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:25.171349 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:25.171325 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:25.172390 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:25.172368 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:13:30.586711 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.586674 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6"] Oct 30 21:13:30.590757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.590736 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.618482 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.618453 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6"] Oct 30 21:13:30.620885 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.620854 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vddk\" (UniqueName: \"kubernetes.io/projected/eff73851-b5a4-4161-9177-5a06b343cec1-kube-api-access-6vddk\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.621015 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.620910 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-credential-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.621015 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.620972 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eff73851-b5a4-4161-9177-5a06b343cec1-istio-token\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.621015 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.621001 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-workload-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.621247 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.621228 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-workload-certs\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.621299 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.621277 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-istio-data\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.621349 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.621305 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eff73851-b5a4-4161-9177-5a06b343cec1-istiod-ca-cert\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.621349 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.621342 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-istio-envoy\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.621443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.621370 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eff73851-b5a4-4161-9177-5a06b343cec1-istio-podinfo\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.722438 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.722411 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eff73851-b5a4-4161-9177-5a06b343cec1-istio-token\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.722582 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.722446 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-workload-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.722582 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.722513 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-workload-certs\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.722582 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.722537 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-istio-data\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.722582 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.722561 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eff73851-b5a4-4161-9177-5a06b343cec1-istiod-ca-cert\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.722794 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.722594 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-istio-envoy\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.722794 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.722621 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eff73851-b5a4-4161-9177-5a06b343cec1-istio-podinfo\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.722794 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.722653 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vddk\" (UniqueName: \"kubernetes.io/projected/eff73851-b5a4-4161-9177-5a06b343cec1-kube-api-access-6vddk\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.722794 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.722692 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-credential-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.722992 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.722949 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-workload-certs\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.722992 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.722975 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-istio-data\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.723121 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.723061 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-workload-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.723246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.723220 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-credential-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.723622 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.723601 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eff73851-b5a4-4161-9177-5a06b343cec1-istiod-ca-cert\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.725220 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.725200 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-istio-envoy\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.725340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.725322 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eff73851-b5a4-4161-9177-5a06b343cec1-istio-podinfo\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.741463 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.741438 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eff73851-b5a4-4161-9177-5a06b343cec1-istio-token\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.742494 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.742475 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vddk\" (UniqueName: \"kubernetes.io/projected/eff73851-b5a4-4161-9177-5a06b343cec1-kube-api-access-6vddk\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:30.918318 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:30.918288 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:31.240991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:31.240916 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6"] Oct 30 21:13:31.261173 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:13:31.261146 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff73851_b5a4_4161_9177_5a06b343cec1.slice/crio-07dea4ca715e9ca39e463e1a3b43dc1b608f9ada72b9ca52c80ded9f9f6b4eb4 WatchSource:0}: Error finding container 07dea4ca715e9ca39e463e1a3b43dc1b608f9ada72b9ca52c80ded9f9f6b4eb4: Status 404 returned error can't find the container with id 07dea4ca715e9ca39e463e1a3b43dc1b608f9ada72b9ca52c80ded9f9f6b4eb4 Oct 30 21:13:31.264701 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:31.264657 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:31.264851 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:31.264748 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:31.264851 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:31.264809 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:32.253578 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:32.253536 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" event={"ID":"eff73851-b5a4-4161-9177-5a06b343cec1","Type":"ContainerStarted","Data":"b26e1d10856de67bb2c4d2cb04f2f9f2e67b73d7b676352d5deda36b82d22a2e"} Oct 30 21:13:32.253973 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:32.253613 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" event={"ID":"eff73851-b5a4-4161-9177-5a06b343cec1","Type":"ContainerStarted","Data":"07dea4ca715e9ca39e463e1a3b43dc1b608f9ada72b9ca52c80ded9f9f6b4eb4"} Oct 30 21:13:32.334213 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:32.334157 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" podStartSLOduration=2.334144264 podStartE2EDuration="2.334144264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:32.322630353 +0000 UTC m=+799.443704098" watchObservedRunningTime="2026-10-30 21:13:32.334144264 +0000 UTC m=+799.455218009" Oct 30 21:13:32.918610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:32.918575 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:32.920404 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:32.920372 4926 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 21:13:32.920520 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:32.920426 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" podUID="eff73851-b5a4-4161-9177-5a06b343cec1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 21:13:33.923551 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:33.923513 4926 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 21:13:33.924196 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:33.924156 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" podUID="eff73851-b5a4-4161-9177-5a06b343cec1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 21:13:34.919264 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:34.919228 4926 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 21:13:34.919536 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:34.919507 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" podUID="eff73851-b5a4-4161-9177-5a06b343cec1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 21:13:35.919194 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:35.919155 4926 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 21:13:35.919655 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:35.919220 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" podUID="eff73851-b5a4-4161-9177-5a06b343cec1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 21:13:36.919615 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:36.919578 4926 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Oct 30 21:13:36.920018 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:36.919649 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" podUID="eff73851-b5a4-4161-9177-5a06b343cec1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Oct 30 21:13:37.493576 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:13:37.493531 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:13:37.702522 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:37.702480 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:13:37.925405 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:37.925371 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:37.925777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:37.925694 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:37.926521 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:37.926500 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:13:41.204209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.204175 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr"] Oct 30 21:13:41.209784 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.209757 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:41.228822 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.228792 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr"] Oct 30 21:13:41.231806 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.231786 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-43c070\"/\"distinct-w5-43c070-kserve-self-signed-certs\"" Oct 30 21:13:41.278790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.278758 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-model-cache\") pod \"distinct-w5-43c070-kserve-79577d4b57-f5hhr\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:41.278970 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.278807 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-dshm\") pod \"distinct-w5-43c070-kserve-79577d4b57-f5hhr\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:41.278970 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.278913 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-home\") pod \"distinct-w5-43c070-kserve-79577d4b57-f5hhr\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:41.279073 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.278993 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-tmp-dir\") pod \"distinct-w5-43c070-kserve-79577d4b57-f5hhr\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:41.279073 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.279043 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jn7l\" (UniqueName: \"kubernetes.io/projected/af79cf23-fc31-4666-842f-5af27a17cb02-kube-api-access-5jn7l\") pod \"distinct-w5-43c070-kserve-79577d4b57-f5hhr\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:41.279073 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.279069 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af79cf23-fc31-4666-842f-5af27a17cb02-tls-certs\") pod \"distinct-w5-43c070-kserve-79577d4b57-f5hhr\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:41.380409 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.380377 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-tmp-dir\") pod \"distinct-w5-43c070-kserve-79577d4b57-f5hhr\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:41.380557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.380426 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jn7l\" (UniqueName: \"kubernetes.io/projected/af79cf23-fc31-4666-842f-5af27a17cb02-kube-api-access-5jn7l\") pod \"distinct-w5-43c070-kserve-79577d4b57-f5hhr\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:41.380557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.380462 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af79cf23-fc31-4666-842f-5af27a17cb02-tls-certs\") pod \"distinct-w5-43c070-kserve-79577d4b57-f5hhr\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:41.380708 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.380633 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-model-cache\") pod \"distinct-w5-43c070-kserve-79577d4b57-f5hhr\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:41.380708 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.380683 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-dshm\") pod \"distinct-w5-43c070-kserve-79577d4b57-f5hhr\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:41.380826 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.380749 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-home\") pod \"distinct-w5-43c070-kserve-79577d4b57-f5hhr\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:41.380889 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.380878 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-tmp-dir\") pod \"distinct-w5-43c070-kserve-79577d4b57-f5hhr\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:41.381022 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.380990 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-model-cache\") pod \"distinct-w5-43c070-kserve-79577d4b57-f5hhr\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:41.381187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.381136 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-home\") pod \"distinct-w5-43c070-kserve-79577d4b57-f5hhr\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:41.383489 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.383467 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-dshm\") pod \"distinct-w5-43c070-kserve-79577d4b57-f5hhr\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:41.383911 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.383885 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af79cf23-fc31-4666-842f-5af27a17cb02-tls-certs\") pod \"distinct-w5-43c070-kserve-79577d4b57-f5hhr\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:41.404876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.400265 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jn7l\" (UniqueName: \"kubernetes.io/projected/af79cf23-fc31-4666-842f-5af27a17cb02-kube-api-access-5jn7l\") pod \"distinct-w5-43c070-kserve-79577d4b57-f5hhr\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:41.529380 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.529286 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:41.837260 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:41.837233 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr"] Oct 30 21:13:41.838368 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:13:41.838340 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf79cf23_fc31_4666_842f_5af27a17cb02.slice/crio-263c3d96939cead756b31c57135b80fe89876916f984286623b5878cc01e6f40 WatchSource:0}: Error finding container 263c3d96939cead756b31c57135b80fe89876916f984286623b5878cc01e6f40: Status 404 returned error can't find the container with id 263c3d96939cead756b31c57135b80fe89876916f984286623b5878cc01e6f40 Oct 30 21:13:42.363984 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:42.363940 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" event={"ID":"af79cf23-fc31-4666-842f-5af27a17cb02","Type":"ContainerStarted","Data":"421dc69ae5bde6099528b1c50477a192de6575bccfc5aa44cbb5f43ac536bb33"} Oct 30 21:13:42.363984 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:42.363987 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" event={"ID":"af79cf23-fc31-4666-842f-5af27a17cb02","Type":"ContainerStarted","Data":"263c3d96939cead756b31c57135b80fe89876916f984286623b5878cc01e6f40"} Oct 30 21:13:42.401504 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:42.401456 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podStartSLOduration=1.401439115 podStartE2EDuration="1.401439115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:42.401044429 +0000 UTC m=+809.522118168" watchObservedRunningTime="2026-10-30 21:13:42.401439115 +0000 UTC m=+809.522512860" Oct 30 21:13:42.529632 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:42.529599 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:45.409420 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.409380 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm"] Oct 30 21:13:45.415493 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.414901 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.440456 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.440425 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm"] Oct 30 21:13:45.529203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.529171 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-workload-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.529203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.529204 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-envoy\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.529443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.529226 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-podinfo\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.529443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.529312 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5397fa31-83d0-4500-94bb-16e1848efe2f-istiod-ca-cert\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.529443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.529361 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-token\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.529597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.529449 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-data\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.529597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.529508 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-workload-certs\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.529695 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.529646 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9895q\" (UniqueName: \"kubernetes.io/projected/5397fa31-83d0-4500-94bb-16e1848efe2f-kube-api-access-9895q\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.529695 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.529692 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-credential-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.631264 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.631229 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-workload-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.631264 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.631262 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-envoy\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.631501 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.631285 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-podinfo\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.631501 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.631321 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5397fa31-83d0-4500-94bb-16e1848efe2f-istiod-ca-cert\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.631501 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.631339 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-token\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.631501 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.631377 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-data\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.631725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.631461 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-workload-certs\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.631815 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.631693 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-workload-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.631912 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.631879 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-data\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.631979 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.631829 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-workload-certs\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.632028 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.631976 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9895q\" (UniqueName: \"kubernetes.io/projected/5397fa31-83d0-4500-94bb-16e1848efe2f-kube-api-access-9895q\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.632104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.632064 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-credential-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.632181 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.632161 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5397fa31-83d0-4500-94bb-16e1848efe2f-istiod-ca-cert\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.632411 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.632387 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-credential-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.633883 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.633856 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-envoy\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.634012 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.633995 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-podinfo\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.649710 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.649687 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-token\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.650065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.650047 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9895q\" (UniqueName: \"kubernetes.io/projected/5397fa31-83d0-4500-94bb-16e1848efe2f-kube-api-access-9895q\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:45.742523 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:45.742496 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:46.063435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:46.063408 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm"] Oct 30 21:13:46.066164 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:13:46.066136 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5397fa31_83d0_4500_94bb_16e1848efe2f.slice/crio-5aa4eedd22596ebba2d1ba747d8b44c2a08c606a66622df46dab325cc33e0f89 WatchSource:0}: Error finding container 5aa4eedd22596ebba2d1ba747d8b44c2a08c606a66622df46dab325cc33e0f89: Status 404 returned error can't find the container with id 5aa4eedd22596ebba2d1ba747d8b44c2a08c606a66622df46dab325cc33e0f89 Oct 30 21:13:46.068958 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:46.068926 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:46.069045 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:46.069001 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:46.069093 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:46.069058 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:13:46.404481 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:46.404394 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" event={"ID":"5397fa31-83d0-4500-94bb-16e1848efe2f","Type":"ContainerStarted","Data":"8f4150dc8404f0a64af3b1ab3e55246bffbfcf2b27670a8c6241b501b063ce4d"} Oct 30 21:13:46.404945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:46.404909 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" event={"ID":"5397fa31-83d0-4500-94bb-16e1848efe2f","Type":"ContainerStarted","Data":"5aa4eedd22596ebba2d1ba747d8b44c2a08c606a66622df46dab325cc33e0f89"} Oct 30 21:13:46.460854 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:46.460793 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" podStartSLOduration=1.460775124 podStartE2EDuration="1.460775124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:13:46.459164203 +0000 UTC m=+813.580237951" watchObservedRunningTime="2026-10-30 21:13:46.460775124 +0000 UTC m=+813.581848871" Oct 30 21:13:46.743057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:46.742990 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:47.755061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:47.754167 4926 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:13:47.755061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:47.754228 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" podUID="5397fa31-83d0-4500-94bb-16e1848efe2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:13:48.744055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:48.743403 4926 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:13:48.744055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:48.743472 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" podUID="5397fa31-83d0-4500-94bb-16e1848efe2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:13:49.743627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:49.743587 4926 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:13:49.744017 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:49.743653 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" podUID="5397fa31-83d0-4500-94bb-16e1848efe2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:13:50.742879 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:50.742844 4926 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:13:50.743050 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:50.742894 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" podUID="5397fa31-83d0-4500-94bb-16e1848efe2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:13:51.529861 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:51.529814 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:51.543098 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:51.543072 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:51.742880 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:51.742847 4926 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 21:13:51.743057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:51.742897 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" podUID="5397fa31-83d0-4500-94bb-16e1848efe2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 21:13:52.469081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:52.469022 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:13:53.056307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:53.056277 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:53.057240 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:53.057206 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:53.058343 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:53.058322 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:13:58.451728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:13:58.451688 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:14:06.403717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.403679 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf"] Oct 30 21:14:06.409670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.408964 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:06.429130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.429105 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ed088d\"/\"distinct-2-w6-ed088d-kserve-self-signed-certs\"" Oct 30 21:14:06.446364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.446325 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf"] Oct 30 21:14:06.450127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.450102 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-home\") pod \"distinct-2-w6-ed088d-kserve-646df6bd65-62dsf\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:06.450414 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.450393 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-dshm\") pod \"distinct-2-w6-ed088d-kserve-646df6bd65-62dsf\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:06.450581 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.450557 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-model-cache\") pod \"distinct-2-w6-ed088d-kserve-646df6bd65-62dsf\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:06.450742 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.450716 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-tls-certs\") pod \"distinct-2-w6-ed088d-kserve-646df6bd65-62dsf\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:06.451010 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.450986 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-tmp-dir\") pod \"distinct-2-w6-ed088d-kserve-646df6bd65-62dsf\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:06.451111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.451023 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhx4h\" (UniqueName: \"kubernetes.io/projected/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-kube-api-access-hhx4h\") pod \"distinct-2-w6-ed088d-kserve-646df6bd65-62dsf\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:06.552465 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.552419 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-tmp-dir\") pod \"distinct-2-w6-ed088d-kserve-646df6bd65-62dsf\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:06.552698 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.552474 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhx4h\" (UniqueName: \"kubernetes.io/projected/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-kube-api-access-hhx4h\") pod \"distinct-2-w6-ed088d-kserve-646df6bd65-62dsf\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:06.552698 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.552555 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-home\") pod \"distinct-2-w6-ed088d-kserve-646df6bd65-62dsf\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:06.552826 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.552687 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-dshm\") pod \"distinct-2-w6-ed088d-kserve-646df6bd65-62dsf\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:06.552826 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.552745 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-model-cache\") pod \"distinct-2-w6-ed088d-kserve-646df6bd65-62dsf\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:06.552935 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.552862 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-tls-certs\") pod \"distinct-2-w6-ed088d-kserve-646df6bd65-62dsf\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:06.552997 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.552928 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-tmp-dir\") pod \"distinct-2-w6-ed088d-kserve-646df6bd65-62dsf\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:06.552997 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.552958 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-home\") pod \"distinct-2-w6-ed088d-kserve-646df6bd65-62dsf\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:06.553251 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.553231 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-model-cache\") pod \"distinct-2-w6-ed088d-kserve-646df6bd65-62dsf\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:06.555237 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.555208 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-dshm\") pod \"distinct-2-w6-ed088d-kserve-646df6bd65-62dsf\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:06.556557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.556534 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-tls-certs\") pod \"distinct-2-w6-ed088d-kserve-646df6bd65-62dsf\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:06.572604 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.571805 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhx4h\" (UniqueName: \"kubernetes.io/projected/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-kube-api-access-hhx4h\") pod \"distinct-2-w6-ed088d-kserve-646df6bd65-62dsf\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:06.737455 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:06.737419 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:07.058616 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:07.058582 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf"] Oct 30 21:14:07.060075 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:14:07.060020 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6c3ba5c_c18e_46bd_ae0a_b2eb12e1b72d.slice/crio-07ebc80fb63fd3540a9d144624f0a29d21f120266876dca3701f90125cbe2c6c WatchSource:0}: Error finding container 07ebc80fb63fd3540a9d144624f0a29d21f120266876dca3701f90125cbe2c6c: Status 404 returned error can't find the container with id 07ebc80fb63fd3540a9d144624f0a29d21f120266876dca3701f90125cbe2c6c Oct 30 21:14:07.494889 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:14:07.494848 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:14:07.600275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:07.600234 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" event={"ID":"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d","Type":"ContainerStarted","Data":"8c0468af1520fe80e8b425f4525176680c01c0fbab25e68cdaef2e64f02c681a"} Oct 30 21:14:07.600469 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:07.600287 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" event={"ID":"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d","Type":"ContainerStarted","Data":"07ebc80fb63fd3540a9d144624f0a29d21f120266876dca3701f90125cbe2c6c"} Oct 30 21:14:07.640783 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:07.640727 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podStartSLOduration=1.640708831 podStartE2EDuration="1.640708831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:07.636784951 +0000 UTC m=+834.757858708" watchObservedRunningTime="2026-10-30 21:14:07.640708831 +0000 UTC m=+834.761782578" Oct 30 21:14:07.737913 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:07.737878 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:15.627735 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.627690 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc"] Oct 30 21:14:15.653058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.646594 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.672109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.663692 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc"] Oct 30 21:14:15.731902 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.731860 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-podinfo\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.731902 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.731895 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-token\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.732210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.732005 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-workload-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.732210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.732078 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-envoy\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.732210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.732195 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-credential-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.732362 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.732232 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-data\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.732438 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.732402 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istiod-ca-cert\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.732596 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.732550 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l858\" (UniqueName: \"kubernetes.io/projected/b4688f01-eef1-4e8f-9370-05fbecbd6f29-kube-api-access-9l858\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.732660 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.732622 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-workload-certs\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.833363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.833327 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l858\" (UniqueName: \"kubernetes.io/projected/b4688f01-eef1-4e8f-9370-05fbecbd6f29-kube-api-access-9l858\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.833568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.833388 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-workload-certs\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.833568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.833423 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-podinfo\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.833568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.833445 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-token\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.833568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.833498 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-workload-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.833568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.833547 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-envoy\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.833831 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.833598 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-credential-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.833831 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.833634 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-data\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.833831 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.833723 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istiod-ca-cert\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.834250 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.834218 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-credential-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.834392 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.834254 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-workload-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.834500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.834479 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-data\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.834555 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.834529 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istiod-ca-cert\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.834739 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.834719 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-workload-certs\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.836087 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.836060 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-envoy\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.837912 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.837890 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-podinfo\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.852128 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.852098 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-token\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.852619 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.852596 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l858\" (UniqueName: \"kubernetes.io/projected/b4688f01-eef1-4e8f-9370-05fbecbd6f29-kube-api-access-9l858\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:15.989479 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:15.989444 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:16.314283 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:14:16.314245 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4688f01_eef1_4e8f_9370_05fbecbd6f29.slice/crio-ad8a3be0a8f71d0723509f0e65a2ae05887a5d8b302798d0600a8c935547e792 WatchSource:0}: Error finding container ad8a3be0a8f71d0723509f0e65a2ae05887a5d8b302798d0600a8c935547e792: Status 404 returned error can't find the container with id ad8a3be0a8f71d0723509f0e65a2ae05887a5d8b302798d0600a8c935547e792 Oct 30 21:14:16.314538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:16.314510 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc"] Oct 30 21:14:16.317135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:16.317097 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:14:16.317239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:16.317181 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:14:16.317239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:16.317224 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:14:16.726298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:16.726262 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" event={"ID":"b4688f01-eef1-4e8f-9370-05fbecbd6f29","Type":"ContainerStarted","Data":"6d1357db6dfd71442df6f57b27ee4b780b554ae4a10551396661ffac4691514e"} Oct 30 21:14:16.726726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:16.726323 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" event={"ID":"b4688f01-eef1-4e8f-9370-05fbecbd6f29","Type":"ContainerStarted","Data":"ad8a3be0a8f71d0723509f0e65a2ae05887a5d8b302798d0600a8c935547e792"} Oct 30 21:14:16.737924 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:16.737896 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:16.751608 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:16.751569 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:16.896610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:16.896543 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" podStartSLOduration=1.896529396 podStartE2EDuration="1.896529396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:16.823181509 +0000 UTC m=+843.944255255" watchObservedRunningTime="2026-10-30 21:14:16.896529396 +0000 UTC m=+844.017603142" Oct 30 21:14:17.005139 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:16.991552 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:17.744873 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:17.744837 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:14:17.994723 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:17.994685 4926 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:14:17.995016 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:17.994957 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" podUID="b4688f01-eef1-4e8f-9370-05fbecbd6f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 21:14:19.001820 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:19.001787 4926 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:14:19.002450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:19.002418 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" podUID="b4688f01-eef1-4e8f-9370-05fbecbd6f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 21:14:20.001084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:19.995412 4926 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:14:20.001084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:19.995474 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" podUID="b4688f01-eef1-4e8f-9370-05fbecbd6f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 21:14:20.991424 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:20.991387 4926 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:14:20.992129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:20.992093 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" podUID="b4688f01-eef1-4e8f-9370-05fbecbd6f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 21:14:21.990445 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:21.990403 4926 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:14:21.990747 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:21.990712 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" podUID="b4688f01-eef1-4e8f-9370-05fbecbd6f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 21:14:22.990653 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:22.990608 4926 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 21:14:22.991074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:22.990681 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" podUID="b4688f01-eef1-4e8f-9370-05fbecbd6f29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 21:14:23.993967 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:23.993935 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:23.994473 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:23.994347 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:23.995168 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:23.995120 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:14:30.598244 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.598146 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj"] Oct 30 21:14:30.602445 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.602421 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.641020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.640993 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj"] Oct 30 21:14:30.667759 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.667725 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-data\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.667953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.667934 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-credential-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.668074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.667965 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-token\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.668074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.667994 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-workload-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.668725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.668260 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-envoy\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.668725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.668301 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-podinfo\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.668725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.668329 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-workload-certs\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.668725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.668393 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a6c48f2-e476-4ae9-82b4-b476a225350d-istiod-ca-cert\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.668725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.668441 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-856mh\" (UniqueName: \"kubernetes.io/projected/4a6c48f2-e476-4ae9-82b4-b476a225350d-kube-api-access-856mh\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.769825 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.769785 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-data\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.770012 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.769834 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-credential-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.770012 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.769864 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-token\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.770012 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.769891 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-workload-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.770012 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.769965 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-envoy\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.770012 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.769991 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-podinfo\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.770324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.770019 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-workload-certs\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.770324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.770099 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a6c48f2-e476-4ae9-82b4-b476a225350d-istiod-ca-cert\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.770324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.770200 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-856mh\" (UniqueName: \"kubernetes.io/projected/4a6c48f2-e476-4ae9-82b4-b476a225350d-kube-api-access-856mh\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.771188 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.771153 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-data\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.771566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.771540 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-workload-certs\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.771807 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.771783 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a6c48f2-e476-4ae9-82b4-b476a225350d-istiod-ca-cert\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.772313 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.772286 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-credential-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.775897 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.775876 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-envoy\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.779998 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.779975 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-podinfo\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.783674 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.772564 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-workload-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.819271 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.819205 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-856mh\" (UniqueName: \"kubernetes.io/projected/4a6c48f2-e476-4ae9-82b4-b476a225350d-kube-api-access-856mh\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.829454 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.829422 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-token\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:30.945652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:30.945619 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:31.282425 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:31.282389 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj"] Oct 30 21:14:31.283240 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:14:31.283211 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a6c48f2_e476_4ae9_82b4_b476a225350d.slice/crio-cfb0c043ab223cbeb15845e4c7ceb6679c680f3e9a1dba0a8e6dabc49371cba9 WatchSource:0}: Error finding container cfb0c043ab223cbeb15845e4c7ceb6679c680f3e9a1dba0a8e6dabc49371cba9: Status 404 returned error can't find the container with id cfb0c043ab223cbeb15845e4c7ceb6679c680f3e9a1dba0a8e6dabc49371cba9 Oct 30 21:14:31.285977 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:31.285942 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:14:31.286078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:31.286018 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:14:31.286153 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:31.286080 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:14:31.846447 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:31.846396 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" event={"ID":"4a6c48f2-e476-4ae9-82b4-b476a225350d","Type":"ContainerStarted","Data":"8c9373da8610358a4fd85e0238b3633925c6510a2798c31c497d3bee985b870d"} Oct 30 21:14:31.846877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:31.846468 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" event={"ID":"4a6c48f2-e476-4ae9-82b4-b476a225350d","Type":"ContainerStarted","Data":"cfb0c043ab223cbeb15845e4c7ceb6679c680f3e9a1dba0a8e6dabc49371cba9"} Oct 30 21:14:31.921998 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:31.921944 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" podStartSLOduration=1.9219285130000001 podStartE2EDuration="1.921928513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:31.917272973 +0000 UTC m=+859.038346720" watchObservedRunningTime="2026-10-30 21:14:31.921928513 +0000 UTC m=+859.043002258" Oct 30 21:14:31.945758 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:31.945701 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:32.947940 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:32.947909 4926 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:14:32.948354 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:32.947967 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" podUID="4a6c48f2-e476-4ae9-82b4-b476a225350d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 21:14:33.946775 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:33.946738 4926 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:14:33.946977 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:33.946791 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" podUID="4a6c48f2-e476-4ae9-82b4-b476a225350d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 21:14:34.018577 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:34.018542 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6426881-gqhqk"] Oct 30 21:14:34.022813 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:34.022786 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6426881-gqhqk" Oct 30 21:14:34.038832 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:34.038806 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6426881-gqhqk"] Oct 30 21:14:34.045621 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:34.045596 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:14:34.045742 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:34.045690 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:14:34.137950 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:34.137913 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a54c7ae-6d2e-4afb-93af-7f712c7995b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6426881-gqhqk\" (UID: \"5a54c7ae-6d2e-4afb-93af-7f712c7995b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6426881-gqhqk" Oct 30 21:14:34.138154 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:34.137992 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rzfm\" (UniqueName: \"kubernetes.io/projected/5a54c7ae-6d2e-4afb-93af-7f712c7995b6-kube-api-access-9rzfm\") pod \"maas-api-revoke-sub-a6426881-gqhqk\" (UID: \"5a54c7ae-6d2e-4afb-93af-7f712c7995b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6426881-gqhqk" Oct 30 21:14:34.138226 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:34.138204 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a54c7ae-6d2e-4afb-93af-7f712c7995b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6426881-gqhqk\" (UID: \"5a54c7ae-6d2e-4afb-93af-7f712c7995b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6426881-gqhqk" Oct 30 21:14:34.239008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:34.238920 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a54c7ae-6d2e-4afb-93af-7f712c7995b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6426881-gqhqk\" (UID: \"5a54c7ae-6d2e-4afb-93af-7f712c7995b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6426881-gqhqk" Oct 30 21:14:34.239008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:34.238984 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a54c7ae-6d2e-4afb-93af-7f712c7995b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6426881-gqhqk\" (UID: \"5a54c7ae-6d2e-4afb-93af-7f712c7995b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6426881-gqhqk" Oct 30 21:14:34.239221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:34.239068 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rzfm\" (UniqueName: \"kubernetes.io/projected/5a54c7ae-6d2e-4afb-93af-7f712c7995b6-kube-api-access-9rzfm\") pod \"maas-api-revoke-sub-a6426881-gqhqk\" (UID: \"5a54c7ae-6d2e-4afb-93af-7f712c7995b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6426881-gqhqk" Oct 30 21:14:34.239623 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:34.239600 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a54c7ae-6d2e-4afb-93af-7f712c7995b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6426881-gqhqk\" (UID: \"5a54c7ae-6d2e-4afb-93af-7f712c7995b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6426881-gqhqk" Oct 30 21:14:34.241576 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:34.241549 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a54c7ae-6d2e-4afb-93af-7f712c7995b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6426881-gqhqk\" (UID: \"5a54c7ae-6d2e-4afb-93af-7f712c7995b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6426881-gqhqk" Oct 30 21:14:34.257291 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:34.257259 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rzfm\" (UniqueName: \"kubernetes.io/projected/5a54c7ae-6d2e-4afb-93af-7f712c7995b6-kube-api-access-9rzfm\") pod \"maas-api-revoke-sub-a6426881-gqhqk\" (UID: \"5a54c7ae-6d2e-4afb-93af-7f712c7995b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6426881-gqhqk" Oct 30 21:14:34.338327 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:34.338288 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6426881-gqhqk" Oct 30 21:14:34.716739 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:34.715178 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6426881-gqhqk"] Oct 30 21:14:34.718838 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:14:34.718809 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a54c7ae_6d2e_4afb_93af_7f712c7995b6.slice/crio-c750b708150faad8f013e558aeda95e70bab176892db81815101c0c55e76db71 WatchSource:0}: Error finding container c750b708150faad8f013e558aeda95e70bab176892db81815101c0c55e76db71: Status 404 returned error can't find the container with id c750b708150faad8f013e558aeda95e70bab176892db81815101c0c55e76db71 Oct 30 21:14:34.901161 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:34.894495 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6426881-gqhqk" event={"ID":"5a54c7ae-6d2e-4afb-93af-7f712c7995b6","Type":"ContainerStarted","Data":"c750b708150faad8f013e558aeda95e70bab176892db81815101c0c55e76db71"} Oct 30 21:14:34.946892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:34.946864 4926 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:14:34.947059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:34.946910 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" podUID="4a6c48f2-e476-4ae9-82b4-b476a225350d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 21:14:35.909537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:35.909498 4926 generic.go:356] "Generic (PLEG): container finished" podID="5a54c7ae-6d2e-4afb-93af-7f712c7995b6" containerID="31c95b5888ed910a6a42deecd69621ceb7f02166552c97c29b4a7f42f61f8d03" exitCode=0 Oct 30 21:14:35.910062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:35.909587 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6426881-gqhqk" event={"ID":"5a54c7ae-6d2e-4afb-93af-7f712c7995b6","Type":"ContainerDied","Data":"31c95b5888ed910a6a42deecd69621ceb7f02166552c97c29b4a7f42f61f8d03"} Oct 30 21:14:35.946133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:35.946100 4926 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:14:35.946320 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:35.946150 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" podUID="4a6c48f2-e476-4ae9-82b4-b476a225350d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 21:14:36.946200 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:36.946173 4926 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:14:36.946613 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:36.946218 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" podUID="4a6c48f2-e476-4ae9-82b4-b476a225350d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 21:14:37.146066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:37.146009 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6426881-gqhqk" Oct 30 21:14:37.325864 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:37.325771 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a54c7ae-6d2e-4afb-93af-7f712c7995b6-kube-api-access-9rzfm\" (UniqueName: \"kubernetes.io/projected/5a54c7ae-6d2e-4afb-93af-7f712c7995b6-kube-api-access-9rzfm\") pod \"5a54c7ae-6d2e-4afb-93af-7f712c7995b6\" (UID: \"5a54c7ae-6d2e-4afb-93af-7f712c7995b6\") " Oct 30 21:14:37.326059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:37.325967 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a54c7ae-6d2e-4afb-93af-7f712c7995b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a54c7ae-6d2e-4afb-93af-7f712c7995b6-maas-api-xks-ca\") pod \"5a54c7ae-6d2e-4afb-93af-7f712c7995b6\" (UID: \"5a54c7ae-6d2e-4afb-93af-7f712c7995b6\") " Oct 30 21:14:37.326059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:37.326008 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a54c7ae-6d2e-4afb-93af-7f712c7995b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a54c7ae-6d2e-4afb-93af-7f712c7995b6-maas-api-openshift-ca\") pod \"5a54c7ae-6d2e-4afb-93af-7f712c7995b6\" (UID: \"5a54c7ae-6d2e-4afb-93af-7f712c7995b6\") " Oct 30 21:14:37.326496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:37.326464 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a54c7ae-6d2e-4afb-93af-7f712c7995b6-maas-api-openshift-ca" pod "5a54c7ae-6d2e-4afb-93af-7f712c7995b6" (UID: "5a54c7ae-6d2e-4afb-93af-7f712c7995b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:14:37.328103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:37.328042 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a54c7ae-6d2e-4afb-93af-7f712c7995b6-maas-api-xks-ca" pod "5a54c7ae-6d2e-4afb-93af-7f712c7995b6" (UID: "5a54c7ae-6d2e-4afb-93af-7f712c7995b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:14:37.328298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:37.328279 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a54c7ae-6d2e-4afb-93af-7f712c7995b6-kube-api-access-9rzfm" pod "5a54c7ae-6d2e-4afb-93af-7f712c7995b6" (UID: "5a54c7ae-6d2e-4afb-93af-7f712c7995b6"). InnerVolumeSpecName "kube-api-access-9rzfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:37.427739 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:37.427698 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a54c7ae-6d2e-4afb-93af-7f712c7995b6-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:14:37.427739 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:37.427742 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rzfm\" (UniqueName: \"kubernetes.io/projected/5a54c7ae-6d2e-4afb-93af-7f712c7995b6-kube-api-access-9rzfm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:14:37.427947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:37.427759 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a54c7ae-6d2e-4afb-93af-7f712c7995b6-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:14:37.493080 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:14:37.493046 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:14:37.928650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:37.928614 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6426881-gqhqk" event={"ID":"5a54c7ae-6d2e-4afb-93af-7f712c7995b6","Type":"ContainerDied","Data":"c750b708150faad8f013e558aeda95e70bab176892db81815101c0c55e76db71"} Oct 30 21:14:37.928828 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:37.928711 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c750b708150faad8f013e558aeda95e70bab176892db81815101c0c55e76db71" Oct 30 21:14:37.928828 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:37.928627 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6426881-gqhqk" Oct 30 21:14:37.946777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:37.946744 4926 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 21:14:37.947208 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:37.946795 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" podUID="4a6c48f2-e476-4ae9-82b4-b476a225350d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 21:14:37.955435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:37.955409 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps"] Oct 30 21:14:37.956319 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:37.956297 4926 cpu_manager.go:414] "containerMap: removing container" podUID="5a54c7ae-6d2e-4afb-93af-7f712c7995b6" containerName="revoke-keys" Oct 30 21:14:37.956398 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:37.956323 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a54c7ae-6d2e-4afb-93af-7f712c7995b6" containerName="revoke-keys" Oct 30 21:14:37.956492 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:37.956478 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a54c7ae-6d2e-4afb-93af-7f712c7995b6" containerName="revoke-keys" Oct 30 21:14:37.960719 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:37.960689 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:37.975521 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:37.975483 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-43c070\"/\"distinct-2-w5-43c070-kserve-self-signed-certs\"" Oct 30 21:14:37.983434 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:37.983409 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps"] Oct 30 21:14:38.034924 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.034876 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-dshm\") pod \"distinct-2-w5-43c070-kserve-64f96c689d-nthps\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:38.035137 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.034981 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-tmp-dir\") pod \"distinct-2-w5-43c070-kserve-64f96c689d-nthps\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:38.035206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.035142 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-model-cache\") pod \"distinct-2-w5-43c070-kserve-64f96c689d-nthps\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:38.035206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.035170 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21cab79c-3466-4600-983d-d8ea4f9dc0d9-tls-certs\") pod \"distinct-2-w5-43c070-kserve-64f96c689d-nthps\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:38.035206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.035192 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-home\") pod \"distinct-2-w5-43c070-kserve-64f96c689d-nthps\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:38.035374 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.035208 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mffc6\" (UniqueName: \"kubernetes.io/projected/21cab79c-3466-4600-983d-d8ea4f9dc0d9-kube-api-access-mffc6\") pod \"distinct-2-w5-43c070-kserve-64f96c689d-nthps\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:38.136484 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.136437 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-dshm\") pod \"distinct-2-w5-43c070-kserve-64f96c689d-nthps\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:38.136693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.136504 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-tmp-dir\") pod \"distinct-2-w5-43c070-kserve-64f96c689d-nthps\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:38.136693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.136618 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-model-cache\") pod \"distinct-2-w5-43c070-kserve-64f96c689d-nthps\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:38.136693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.136647 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21cab79c-3466-4600-983d-d8ea4f9dc0d9-tls-certs\") pod \"distinct-2-w5-43c070-kserve-64f96c689d-nthps\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:38.136693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.136677 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-home\") pod \"distinct-2-w5-43c070-kserve-64f96c689d-nthps\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:38.136928 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.136702 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mffc6\" (UniqueName: \"kubernetes.io/projected/21cab79c-3466-4600-983d-d8ea4f9dc0d9-kube-api-access-mffc6\") pod \"distinct-2-w5-43c070-kserve-64f96c689d-nthps\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:38.137578 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.137190 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-model-cache\") pod \"distinct-2-w5-43c070-kserve-64f96c689d-nthps\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:38.137578 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.137465 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-tmp-dir\") pod \"distinct-2-w5-43c070-kserve-64f96c689d-nthps\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:38.137779 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.137697 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-home\") pod \"distinct-2-w5-43c070-kserve-64f96c689d-nthps\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:38.140237 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.140211 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21cab79c-3466-4600-983d-d8ea4f9dc0d9-tls-certs\") pod \"distinct-2-w5-43c070-kserve-64f96c689d-nthps\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:38.140759 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.140733 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-dshm\") pod \"distinct-2-w5-43c070-kserve-64f96c689d-nthps\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:38.169988 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.169959 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mffc6\" (UniqueName: \"kubernetes.io/projected/21cab79c-3466-4600-983d-d8ea4f9dc0d9-kube-api-access-mffc6\") pod \"distinct-2-w5-43c070-kserve-64f96c689d-nthps\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:38.285423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.285328 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:38.666514 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.666479 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps"] Oct 30 21:14:38.936586 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.936541 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" event={"ID":"21cab79c-3466-4600-983d-d8ea4f9dc0d9","Type":"ContainerStarted","Data":"ae809aacc93f33f978e99d43178b4be1a8d7c3b3648d5df320b972368e0e0983"} Oct 30 21:14:38.936773 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.936598 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" event={"ID":"21cab79c-3466-4600-983d-d8ea4f9dc0d9","Type":"ContainerStarted","Data":"9a9fcd08ff5bbed67542d69731a85f4189b7ee27389b7af7e98569df8e7468c8"} Oct 30 21:14:38.950639 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.950614 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:38.950975 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.950880 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:38.951710 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.951691 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:14:38.979602 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:38.979541 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podStartSLOduration=1.979524321 podStartE2EDuration="1.979524321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:38.973402438 +0000 UTC m=+866.094476185" watchObservedRunningTime="2026-10-30 21:14:38.979524321 +0000 UTC m=+866.100598068" Oct 30 21:14:39.286927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:39.286845 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:39.382711 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:39.382679 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:14:45.596135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.596100 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw"] Oct 30 21:14:45.601631 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.601601 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.647624 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.641645 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw"] Oct 30 21:14:45.732320 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.732287 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-envoy\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.732320 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.732323 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwzjm\" (UniqueName: \"kubernetes.io/projected/4bbaeb83-ec01-4556-b964-c56981dc1c5c-kube-api-access-jwzjm\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.732524 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.732345 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-token\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.732524 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.732490 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-workload-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.732524 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.732518 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-data\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.732620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.732547 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-podinfo\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.732680 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.732658 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-credential-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.732760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.732718 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-workload-certs\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.732760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.732754 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istiod-ca-cert\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.833993 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.833961 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-workload-certs\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.833993 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.834001 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istiod-ca-cert\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.834265 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.834090 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-envoy\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.834265 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.834115 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwzjm\" (UniqueName: \"kubernetes.io/projected/4bbaeb83-ec01-4556-b964-c56981dc1c5c-kube-api-access-jwzjm\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.834430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.834265 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-token\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.834430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.834389 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-workload-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.834430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.834418 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-data\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.834430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.834419 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-workload-certs\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.834677 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.834458 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-podinfo\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.834677 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.834552 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-credential-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.834677 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.834663 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-workload-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.834831 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.834720 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-data\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.834892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.834853 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istiod-ca-cert\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.834985 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.834962 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-credential-socket\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.836935 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.836909 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-envoy\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.837161 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.837144 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-podinfo\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.854769 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.854710 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-token\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.854865 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.854772 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwzjm\" (UniqueName: \"kubernetes.io/projected/4bbaeb83-ec01-4556-b964-c56981dc1c5c-kube-api-access-jwzjm\") pod \"e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:45.948755 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:45.948724 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:46.280374 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:46.280340 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5"] Oct 30 21:14:46.286462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:46.286439 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5" Oct 30 21:14:46.302166 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:46.299435 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw"] Oct 30 21:14:46.302166 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:46.301211 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:14:46.305373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:46.304648 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:14:46.306649 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:14:46.306621 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bbaeb83_ec01_4556_b964_c56981dc1c5c.slice/crio-0bfaadc3525203145bccbdf2cccb72c621616360515c1c1f2b0b920af3bc2d22 WatchSource:0}: Error finding container 0bfaadc3525203145bccbdf2cccb72c621616360515c1c1f2b0b920af3bc2d22: Status 404 returned error can't find the container with id 0bfaadc3525203145bccbdf2cccb72c621616360515c1c1f2b0b920af3bc2d22 Oct 30 21:14:46.308763 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:46.308739 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5"] Oct 30 21:14:46.309951 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:46.309917 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:14:46.310069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:46.309995 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:14:46.310069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:46.310052 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:14:46.351192 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:46.351152 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492f0633-33e2-4c68-8ac5-342fc9a84db8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a7ae484-q7jn5\" (UID: \"492f0633-33e2-4c68-8ac5-342fc9a84db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5" Oct 30 21:14:46.351304 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:46.351246 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492f0633-33e2-4c68-8ac5-342fc9a84db8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a7ae484-q7jn5\" (UID: \"492f0633-33e2-4c68-8ac5-342fc9a84db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5" Oct 30 21:14:46.351304 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:46.351297 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq75r\" (UniqueName: \"kubernetes.io/projected/492f0633-33e2-4c68-8ac5-342fc9a84db8-kube-api-access-rq75r\") pod \"maas-api-revoke-sub-5a7ae484-q7jn5\" (UID: \"492f0633-33e2-4c68-8ac5-342fc9a84db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5" Oct 30 21:14:46.452992 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:46.452955 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492f0633-33e2-4c68-8ac5-342fc9a84db8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a7ae484-q7jn5\" (UID: \"492f0633-33e2-4c68-8ac5-342fc9a84db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5" Oct 30 21:14:46.453186 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:46.452999 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492f0633-33e2-4c68-8ac5-342fc9a84db8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a7ae484-q7jn5\" (UID: \"492f0633-33e2-4c68-8ac5-342fc9a84db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5" Oct 30 21:14:46.453186 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:46.453025 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq75r\" (UniqueName: \"kubernetes.io/projected/492f0633-33e2-4c68-8ac5-342fc9a84db8-kube-api-access-rq75r\") pod \"maas-api-revoke-sub-5a7ae484-q7jn5\" (UID: \"492f0633-33e2-4c68-8ac5-342fc9a84db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5" Oct 30 21:14:46.453697 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:46.453674 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492f0633-33e2-4c68-8ac5-342fc9a84db8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a7ae484-q7jn5\" (UID: \"492f0633-33e2-4c68-8ac5-342fc9a84db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5" Oct 30 21:14:46.455485 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:46.455461 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492f0633-33e2-4c68-8ac5-342fc9a84db8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a7ae484-q7jn5\" (UID: \"492f0633-33e2-4c68-8ac5-342fc9a84db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5" Oct 30 21:14:46.472013 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:46.471984 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq75r\" (UniqueName: \"kubernetes.io/projected/492f0633-33e2-4c68-8ac5-342fc9a84db8-kube-api-access-rq75r\") pod \"maas-api-revoke-sub-5a7ae484-q7jn5\" (UID: \"492f0633-33e2-4c68-8ac5-342fc9a84db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5" Oct 30 21:14:46.616930 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:46.616780 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5" Oct 30 21:14:46.933040 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:46.933000 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5"] Oct 30 21:14:46.933740 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:14:46.933710 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod492f0633_33e2_4c68_8ac5_342fc9a84db8.slice/crio-6012f6ee990438ab7ab6adf98d68f554a8d5126726e9e3e58b08a704d32059b7 WatchSource:0}: Error finding container 6012f6ee990438ab7ab6adf98d68f554a8d5126726e9e3e58b08a704d32059b7: Status 404 returned error can't find the container with id 6012f6ee990438ab7ab6adf98d68f554a8d5126726e9e3e58b08a704d32059b7 Oct 30 21:14:47.064716 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:47.064683 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5" event={"ID":"492f0633-33e2-4c68-8ac5-342fc9a84db8","Type":"ContainerStarted","Data":"6012f6ee990438ab7ab6adf98d68f554a8d5126726e9e3e58b08a704d32059b7"} Oct 30 21:14:47.066537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:47.066508 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" event={"ID":"4bbaeb83-ec01-4556-b964-c56981dc1c5c","Type":"ContainerStarted","Data":"1313412c3b98d6ec1d26797e636bfa4c90e104b183461de65ddfc278395c2b8d"} Oct 30 21:14:47.066682 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:47.066559 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" event={"ID":"4bbaeb83-ec01-4556-b964-c56981dc1c5c","Type":"ContainerStarted","Data":"0bfaadc3525203145bccbdf2cccb72c621616360515c1c1f2b0b920af3bc2d22"} Oct 30 21:14:47.113649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:47.113588 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" podStartSLOduration=2.113572928 podStartE2EDuration="2.113572928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:47.110419413 +0000 UTC m=+874.231493160" watchObservedRunningTime="2026-10-30 21:14:47.113572928 +0000 UTC m=+874.234646674" Oct 30 21:14:47.949804 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:47.949767 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:47.951403 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:47.951373 4926 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:14:47.951545 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:47.951432 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" podUID="4bbaeb83-ec01-4556-b964-c56981dc1c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:14:48.108184 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:48.108140 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5" event={"ID":"492f0633-33e2-4c68-8ac5-342fc9a84db8","Type":"ContainerStarted","Data":"07c253b136a6dce0b2deb21e241c6e343510bfc35ab4629a440377d6210cb428"} Oct 30 21:14:48.146731 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:48.146674 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5" podStartSLOduration=2.146660467 podStartE2EDuration="2.146660467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:48.145381999 +0000 UTC m=+875.266455745" watchObservedRunningTime="2026-10-30 21:14:48.146660467 +0000 UTC m=+875.267734228" Oct 30 21:14:48.298198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:48.288665 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:48.312480 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:48.312442 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:48.950107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:48.950074 4926 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:14:48.950596 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:48.950126 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" podUID="4bbaeb83-ec01-4556-b964-c56981dc1c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:14:49.119423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:49.119388 4926 generic.go:356] "Generic (PLEG): container finished" podID="492f0633-33e2-4c68-8ac5-342fc9a84db8" containerID="07c253b136a6dce0b2deb21e241c6e343510bfc35ab4629a440377d6210cb428" exitCode=0 Oct 30 21:14:49.119611 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:49.119487 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5" event={"ID":"492f0633-33e2-4c68-8ac5-342fc9a84db8","Type":"ContainerDied","Data":"07c253b136a6dce0b2deb21e241c6e343510bfc35ab4629a440377d6210cb428"} Oct 30 21:14:49.137840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:49.137811 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:14:49.949545 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:49.949515 4926 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:14:49.949756 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:49.949569 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" podUID="4bbaeb83-ec01-4556-b964-c56981dc1c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:14:50.335615 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:50.335342 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5" Oct 30 21:14:50.349145 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:50.349115 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/492f0633-33e2-4c68-8ac5-342fc9a84db8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492f0633-33e2-4c68-8ac5-342fc9a84db8-maas-api-xks-ca\") pod \"492f0633-33e2-4c68-8ac5-342fc9a84db8\" (UID: \"492f0633-33e2-4c68-8ac5-342fc9a84db8\") " Oct 30 21:14:50.349270 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:50.349240 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/492f0633-33e2-4c68-8ac5-342fc9a84db8-kube-api-access-rq75r\" (UniqueName: \"kubernetes.io/projected/492f0633-33e2-4c68-8ac5-342fc9a84db8-kube-api-access-rq75r\") pod \"492f0633-33e2-4c68-8ac5-342fc9a84db8\" (UID: \"492f0633-33e2-4c68-8ac5-342fc9a84db8\") " Oct 30 21:14:50.349391 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:50.349374 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/492f0633-33e2-4c68-8ac5-342fc9a84db8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492f0633-33e2-4c68-8ac5-342fc9a84db8-maas-api-openshift-ca\") pod \"492f0633-33e2-4c68-8ac5-342fc9a84db8\" (UID: \"492f0633-33e2-4c68-8ac5-342fc9a84db8\") " Oct 30 21:14:50.349710 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:50.349685 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/492f0633-33e2-4c68-8ac5-342fc9a84db8-maas-api-openshift-ca" pod "492f0633-33e2-4c68-8ac5-342fc9a84db8" (UID: "492f0633-33e2-4c68-8ac5-342fc9a84db8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:14:50.351240 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:50.351213 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/492f0633-33e2-4c68-8ac5-342fc9a84db8-maas-api-xks-ca" pod "492f0633-33e2-4c68-8ac5-342fc9a84db8" (UID: "492f0633-33e2-4c68-8ac5-342fc9a84db8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:14:50.351338 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:50.351320 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/492f0633-33e2-4c68-8ac5-342fc9a84db8-kube-api-access-rq75r" pod "492f0633-33e2-4c68-8ac5-342fc9a84db8" (UID: "492f0633-33e2-4c68-8ac5-342fc9a84db8"). InnerVolumeSpecName "kube-api-access-rq75r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:14:50.451796 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:50.451759 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/492f0633-33e2-4c68-8ac5-342fc9a84db8-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:14:50.451796 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:50.451799 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/492f0633-33e2-4c68-8ac5-342fc9a84db8-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:14:50.451990 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:50.451816 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq75r\" (UniqueName: \"kubernetes.io/projected/492f0633-33e2-4c68-8ac5-342fc9a84db8-kube-api-access-rq75r\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:14:50.949332 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:50.949296 4926 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:14:50.949509 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:50.949373 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" podUID="4bbaeb83-ec01-4556-b964-c56981dc1c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:14:51.138844 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:51.138819 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5" Oct 30 21:14:51.138996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:51.138816 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5" event={"ID":"492f0633-33e2-4c68-8ac5-342fc9a84db8","Type":"ContainerDied","Data":"6012f6ee990438ab7ab6adf98d68f554a8d5126726e9e3e58b08a704d32059b7"} Oct 30 21:14:51.138996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:51.138949 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6012f6ee990438ab7ab6adf98d68f554a8d5126726e9e3e58b08a704d32059b7" Oct 30 21:14:51.949790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:51.949752 4926 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:14:51.950207 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:51.949827 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" podUID="4bbaeb83-ec01-4556-b964-c56981dc1c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:14:52.949675 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:52.949633 4926 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 21:14:52.949989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:52.949696 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" podUID="4bbaeb83-ec01-4556-b964-c56981dc1c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 21:14:53.142080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.138136 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2"] Oct 30 21:14:53.142080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.139066 4926 cpu_manager.go:414] "containerMap: removing container" podUID="492f0633-33e2-4c68-8ac5-342fc9a84db8" containerName="revoke-keys" Oct 30 21:14:53.142080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.139086 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="492f0633-33e2-4c68-8ac5-342fc9a84db8" containerName="revoke-keys" Oct 30 21:14:53.142080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.139266 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="492f0633-33e2-4c68-8ac5-342fc9a84db8" containerName="revoke-keys" Oct 30 21:14:53.146325 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.146295 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:14:53.163124 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.163100 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-56ac74\"/\"distinct-2-w1-56ac74-kserve-self-signed-certs\"" Oct 30 21:14:53.165887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.165809 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2"] Oct 30 21:14:53.205064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.204962 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-home\") pod \"distinct-2-w1-56ac74-kserve-8b899879f-9nrt2\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:14:53.205064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.204997 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-tls-certs\") pod \"distinct-2-w1-56ac74-kserve-8b899879f-9nrt2\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:14:53.205277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.205166 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-tmp-dir\") pod \"distinct-2-w1-56ac74-kserve-8b899879f-9nrt2\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:14:53.205277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.205267 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk692\" (UniqueName: \"kubernetes.io/projected/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-kube-api-access-nk692\") pod \"distinct-2-w1-56ac74-kserve-8b899879f-9nrt2\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:14:53.205372 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.205302 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-dshm\") pod \"distinct-2-w1-56ac74-kserve-8b899879f-9nrt2\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:14:53.205435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.205370 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-model-cache\") pod \"distinct-2-w1-56ac74-kserve-8b899879f-9nrt2\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:14:53.307157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.307118 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-tmp-dir\") pod \"distinct-2-w1-56ac74-kserve-8b899879f-9nrt2\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:14:53.307368 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.307223 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk692\" (UniqueName: \"kubernetes.io/projected/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-kube-api-access-nk692\") pod \"distinct-2-w1-56ac74-kserve-8b899879f-9nrt2\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:14:53.307368 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.307258 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-dshm\") pod \"distinct-2-w1-56ac74-kserve-8b899879f-9nrt2\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:14:53.307368 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.307323 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-model-cache\") pod \"distinct-2-w1-56ac74-kserve-8b899879f-9nrt2\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:14:53.307559 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.307460 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-home\") pod \"distinct-2-w1-56ac74-kserve-8b899879f-9nrt2\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:14:53.307559 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.307484 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-tls-certs\") pod \"distinct-2-w1-56ac74-kserve-8b899879f-9nrt2\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:14:53.307676 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.307594 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-tmp-dir\") pod \"distinct-2-w1-56ac74-kserve-8b899879f-9nrt2\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:14:53.308176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.308151 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-model-cache\") pod \"distinct-2-w1-56ac74-kserve-8b899879f-9nrt2\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:14:53.308367 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.308209 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-home\") pod \"distinct-2-w1-56ac74-kserve-8b899879f-9nrt2\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:14:53.309936 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.309913 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-dshm\") pod \"distinct-2-w1-56ac74-kserve-8b899879f-9nrt2\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:14:53.312365 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.312340 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-tls-certs\") pod \"distinct-2-w1-56ac74-kserve-8b899879f-9nrt2\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:14:53.333668 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.333642 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk692\" (UniqueName: \"kubernetes.io/projected/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-kube-api-access-nk692\") pod \"distinct-2-w1-56ac74-kserve-8b899879f-9nrt2\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:14:53.470833 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.470760 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:14:53.796344 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:53.795430 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2"] Oct 30 21:14:53.796344 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:14:53.795663 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc449fbe_a012_44a2_b57b_5ab7bbe1f44f.slice/crio-d17d47553fccaceea09b6d2c843a862949237437e241e89537a7bec21811e156 WatchSource:0}: Error finding container d17d47553fccaceea09b6d2c843a862949237437e241e89537a7bec21811e156: Status 404 returned error can't find the container with id d17d47553fccaceea09b6d2c843a862949237437e241e89537a7bec21811e156 Oct 30 21:14:54.127428 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:54.127348 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:54.128974 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:54.128952 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:54.136397 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:54.136376 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:14:54.179747 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:54.179710 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" event={"ID":"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f","Type":"ContainerStarted","Data":"7a04844314a11e178a8ab24fe805478b79cf01f25e0cd2734da35498e2e70a75"} Oct 30 21:14:54.179943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:54.179756 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" event={"ID":"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f","Type":"ContainerStarted","Data":"d17d47553fccaceea09b6d2c843a862949237437e241e89537a7bec21811e156"} Oct 30 21:14:54.223894 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:54.223842 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podStartSLOduration=1.223828151 podStartE2EDuration="1.223828151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:14:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:14:54.222291506 +0000 UTC m=+881.343365253" watchObservedRunningTime="2026-10-30 21:14:54.223828151 +0000 UTC m=+881.344901937" Oct 30 21:14:54.471876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:14:54.471830 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:15:00.157597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.157563 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5"] Oct 30 21:15:00.164241 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.162857 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5" Oct 30 21:15:00.176840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.176810 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm"] Oct 30 21:15:00.204917 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.189734 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm" Oct 30 21:15:00.204917 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.202548 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:15:00.208588 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.208554 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5"] Oct 30 21:15:00.227328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.224309 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c"] Oct 30 21:15:00.229512 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.229488 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c" Oct 30 21:15:00.234383 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.234014 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2"] Oct 30 21:15:00.239903 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.239461 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2" Oct 30 21:15:00.240294 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.240269 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm"] Oct 30 21:15:00.246280 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.246257 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2"] Oct 30 21:15:00.257023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.257000 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c"] Oct 30 21:15:00.313156 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.313113 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zzl7\" (UniqueName: \"kubernetes.io/projected/e5160e9e-5873-438e-9679-7c0227fe753a-kube-api-access-4zzl7\") pod \"maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5\" (UID: \"e5160e9e-5873-438e-9679-7c0227fe753a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5" Oct 30 21:15:00.313350 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.313179 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81c872ad-0ad6-4f1a-93c4-fc79532d5e65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c\" (UID: \"81c872ad-0ad6-4f1a-93c4-fc79532d5e65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c" Oct 30 21:15:00.313428 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.313371 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxc29\" (UniqueName: \"kubernetes.io/projected/52b459ac-7beb-42a9-bcb6-9f193d76632c-kube-api-access-lxc29\") pod \"maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm\" (UID: \"52b459ac-7beb-42a9-bcb6-9f193d76632c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm" Oct 30 21:15:00.313529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.313504 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b459ac-7beb-42a9-bcb6-9f193d76632c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm\" (UID: \"52b459ac-7beb-42a9-bcb6-9f193d76632c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm" Oct 30 21:15:00.313588 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.313570 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5160e9e-5873-438e-9679-7c0227fe753a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5\" (UID: \"e5160e9e-5873-438e-9679-7c0227fe753a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5" Oct 30 21:15:00.313819 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.313652 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtftb\" (UniqueName: \"kubernetes.io/projected/81c872ad-0ad6-4f1a-93c4-fc79532d5e65-kube-api-access-wtftb\") pod \"maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c\" (UID: \"81c872ad-0ad6-4f1a-93c4-fc79532d5e65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c" Oct 30 21:15:00.414589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.414556 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxc29\" (UniqueName: \"kubernetes.io/projected/52b459ac-7beb-42a9-bcb6-9f193d76632c-kube-api-access-lxc29\") pod \"maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm\" (UID: \"52b459ac-7beb-42a9-bcb6-9f193d76632c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm" Oct 30 21:15:00.414776 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.414656 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b459ac-7beb-42a9-bcb6-9f193d76632c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm\" (UID: \"52b459ac-7beb-42a9-bcb6-9f193d76632c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm" Oct 30 21:15:00.414776 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.414690 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5160e9e-5873-438e-9679-7c0227fe753a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5\" (UID: \"e5160e9e-5873-438e-9679-7c0227fe753a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5" Oct 30 21:15:00.414776 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.414737 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qgs6\" (UniqueName: \"kubernetes.io/projected/34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2-kube-api-access-9qgs6\") pod \"maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2\" (UID: \"34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2" Oct 30 21:15:00.414959 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.414792 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2\" (UID: \"34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2" Oct 30 21:15:00.414959 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.414823 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtftb\" (UniqueName: \"kubernetes.io/projected/81c872ad-0ad6-4f1a-93c4-fc79532d5e65-kube-api-access-wtftb\") pod \"maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c\" (UID: \"81c872ad-0ad6-4f1a-93c4-fc79532d5e65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c" Oct 30 21:15:00.414959 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.414914 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zzl7\" (UniqueName: \"kubernetes.io/projected/e5160e9e-5873-438e-9679-7c0227fe753a-kube-api-access-4zzl7\") pod \"maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5\" (UID: \"e5160e9e-5873-438e-9679-7c0227fe753a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5" Oct 30 21:15:00.415189 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.415001 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81c872ad-0ad6-4f1a-93c4-fc79532d5e65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c\" (UID: \"81c872ad-0ad6-4f1a-93c4-fc79532d5e65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c" Oct 30 21:15:00.415573 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.415550 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b459ac-7beb-42a9-bcb6-9f193d76632c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm\" (UID: \"52b459ac-7beb-42a9-bcb6-9f193d76632c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm" Oct 30 21:15:00.415658 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.415586 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5160e9e-5873-438e-9679-7c0227fe753a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5\" (UID: \"e5160e9e-5873-438e-9679-7c0227fe753a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5" Oct 30 21:15:00.415736 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.415719 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81c872ad-0ad6-4f1a-93c4-fc79532d5e65-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c\" (UID: \"81c872ad-0ad6-4f1a-93c4-fc79532d5e65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c" Oct 30 21:15:00.432736 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.432713 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtftb\" (UniqueName: \"kubernetes.io/projected/81c872ad-0ad6-4f1a-93c4-fc79532d5e65-kube-api-access-wtftb\") pod \"maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c\" (UID: \"81c872ad-0ad6-4f1a-93c4-fc79532d5e65\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c" Oct 30 21:15:00.432840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.432743 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxc29\" (UniqueName: \"kubernetes.io/projected/52b459ac-7beb-42a9-bcb6-9f193d76632c-kube-api-access-lxc29\") pod \"maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm\" (UID: \"52b459ac-7beb-42a9-bcb6-9f193d76632c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm" Oct 30 21:15:00.432881 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.432851 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zzl7\" (UniqueName: \"kubernetes.io/projected/e5160e9e-5873-438e-9679-7c0227fe753a-kube-api-access-4zzl7\") pod \"maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5\" (UID: \"e5160e9e-5873-438e-9679-7c0227fe753a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5" Oct 30 21:15:00.486909 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.486880 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5" Oct 30 21:15:00.516582 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.516547 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qgs6\" (UniqueName: \"kubernetes.io/projected/34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2-kube-api-access-9qgs6\") pod \"maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2\" (UID: \"34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2" Oct 30 21:15:00.516727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.516612 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2\" (UID: \"34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2" Oct 30 21:15:00.517375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.517351 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2\" (UID: \"34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2" Oct 30 21:15:00.531148 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.531120 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm" Oct 30 21:15:00.538831 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.538813 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qgs6\" (UniqueName: \"kubernetes.io/projected/34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2-kube-api-access-9qgs6\") pod \"maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2\" (UID: \"34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2" Oct 30 21:15:00.550900 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.550878 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c" Oct 30 21:15:00.583618 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.576982 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2" Oct 30 21:15:00.642165 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.638943 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64"] Oct 30 21:15:00.650072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.649996 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.682864 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.682834 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64"] Oct 30 21:15:00.803672 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.803644 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5"] Oct 30 21:15:00.803808 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:15:00.803787 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5160e9e_5873_438e_9679_7c0227fe753a.slice/crio-a7b60accf6a048e76984747c8dbee69bd9b80475cae127b37095426ac8610f93 WatchSource:0}: Error finding container a7b60accf6a048e76984747c8dbee69bd9b80475cae127b37095426ac8610f93: Status 404 returned error can't find the container with id a7b60accf6a048e76984747c8dbee69bd9b80475cae127b37095426ac8610f93 Oct 30 21:15:00.822952 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.822918 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-workload-certs\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.823081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.822971 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42d62ed-b880-4a25-978c-47c703a5cdd3-istiod-ca-cert\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.823081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.823015 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-data\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.823220 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.823157 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-envoy\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.823220 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.823186 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52pbj\" (UniqueName: \"kubernetes.io/projected/b42d62ed-b880-4a25-978c-47c703a5cdd3-kube-api-access-52pbj\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.823325 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.823295 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-workload-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.823325 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.823313 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-credential-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.823438 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.823372 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-token\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.823577 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.823390 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-podinfo\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.901182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.901149 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm"] Oct 30 21:15:00.924994 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.924929 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-workload-certs\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.924994 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.924982 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42d62ed-b880-4a25-978c-47c703a5cdd3-istiod-ca-cert\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.925186 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.925073 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-data\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.925186 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.925124 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-envoy\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.925186 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.925147 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52pbj\" (UniqueName: \"kubernetes.io/projected/b42d62ed-b880-4a25-978c-47c703a5cdd3-kube-api-access-52pbj\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.925347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.925221 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-workload-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.925347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.925241 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-credential-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.925347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.925309 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-token\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.925347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.925335 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-podinfo\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.925558 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.925449 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-workload-certs\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.925558 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.925522 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-data\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.925790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.925721 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-credential-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.925907 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.925880 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-workload-socket\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.925907 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.925883 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42d62ed-b880-4a25-978c-47c703a5cdd3-istiod-ca-cert\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.965744 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.965724 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c"] Oct 30 21:15:00.967526 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.967499 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-envoy\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.967908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.967886 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-podinfo\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.968114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.968065 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52pbj\" (UniqueName: \"kubernetes.io/projected/b42d62ed-b880-4a25-978c-47c703a5cdd3-kube-api-access-52pbj\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.968636 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.968596 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-token\") pod \"e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:00.969525 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:15:00.969231 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52b459ac_7beb_42a9_bcb6_9f193d76632c.slice/crio-8d4e956a970f7156c198a3f5c16e5d119a32909384f1142ac624d1abf3071fed WatchSource:0}: Error finding container 8d4e956a970f7156c198a3f5c16e5d119a32909384f1142ac624d1abf3071fed: Status 404 returned error can't find the container with id 8d4e956a970f7156c198a3f5c16e5d119a32909384f1142ac624d1abf3071fed Oct 30 21:15:00.971085 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:15:00.971062 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81c872ad_0ad6_4f1a_93c4_fc79532d5e65.slice/crio-9d5a1bee5a69196fd4a3cfa32e4e97abe8ba9b1159bfd08e103370cefeabc8ca WatchSource:0}: Error finding container 9d5a1bee5a69196fd4a3cfa32e4e97abe8ba9b1159bfd08e103370cefeabc8ca: Status 404 returned error can't find the container with id 9d5a1bee5a69196fd4a3cfa32e4e97abe8ba9b1159bfd08e103370cefeabc8ca Oct 30 21:15:00.993704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:00.993678 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:01.032838 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:01.032812 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2"] Oct 30 21:15:01.072439 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:15:01.072404 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f1501c_70e6_4e82_aa6c_ac0e0bbfb0d2.slice/crio-3375a430f4a30dd9a6f3a4315014c02613c0217e884a062768ff40505031d06a WatchSource:0}: Error finding container 3375a430f4a30dd9a6f3a4315014c02613c0217e884a062768ff40505031d06a: Status 404 returned error can't find the container with id 3375a430f4a30dd9a6f3a4315014c02613c0217e884a062768ff40505031d06a Oct 30 21:15:01.243018 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:01.242985 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2" event={"ID":"34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2","Type":"ContainerStarted","Data":"3375a430f4a30dd9a6f3a4315014c02613c0217e884a062768ff40505031d06a"} Oct 30 21:15:01.244832 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:01.244799 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5" event={"ID":"e5160e9e-5873-438e-9679-7c0227fe753a","Type":"ContainerStarted","Data":"0d38cd0ecbbfe950db8270db48ef3c1b75941c13e43ba56f2cdfcf546e8cad09"} Oct 30 21:15:01.244988 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:01.244855 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5" event={"ID":"e5160e9e-5873-438e-9679-7c0227fe753a","Type":"ContainerStarted","Data":"a7b60accf6a048e76984747c8dbee69bd9b80475cae127b37095426ac8610f93"} Oct 30 21:15:01.246172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:01.246141 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c" event={"ID":"81c872ad-0ad6-4f1a-93c4-fc79532d5e65","Type":"ContainerStarted","Data":"9d5a1bee5a69196fd4a3cfa32e4e97abe8ba9b1159bfd08e103370cefeabc8ca"} Oct 30 21:15:01.247298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:01.247277 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm" event={"ID":"52b459ac-7beb-42a9-bcb6-9f193d76632c","Type":"ContainerStarted","Data":"8d4e956a970f7156c198a3f5c16e5d119a32909384f1142ac624d1abf3071fed"} Oct 30 21:15:01.281210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:01.281167 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5" podStartSLOduration=1.281154671 podStartE2EDuration="1.281154671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:01.280473764 +0000 UTC m=+888.401547509" watchObservedRunningTime="2026-10-30 21:15:01.281154671 +0000 UTC m=+888.402228416" Oct 30 21:15:01.357177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:01.357152 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64"] Oct 30 21:15:01.369140 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:15:01.369096 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42d62ed_b880_4a25_978c_47c703a5cdd3.slice/crio-b2c6af419e6768ff6f95627b06e6776daf8c5526183ecaaa2d1eb4e73f6b3055 WatchSource:0}: Error finding container b2c6af419e6768ff6f95627b06e6776daf8c5526183ecaaa2d1eb4e73f6b3055: Status 404 returned error can't find the container with id b2c6af419e6768ff6f95627b06e6776daf8c5526183ecaaa2d1eb4e73f6b3055 Oct 30 21:15:01.372401 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:01.372360 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:01.372503 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:01.372458 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:01.372552 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:01.372500 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:02.257018 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:02.256980 4926 generic.go:356] "Generic (PLEG): container finished" podID="34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2" containerID="366fc18ce3b3f94145556568ce48d671a9ed8685e053edc15d84b2eb1b0ce90e" exitCode=60 Oct 30 21:15:02.257493 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:02.257064 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2" event={"ID":"34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2","Type":"ContainerDied","Data":"366fc18ce3b3f94145556568ce48d671a9ed8685e053edc15d84b2eb1b0ce90e"} Oct 30 21:15:02.257493 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:02.257390 4926 scope.go:122] "RemoveContainer" containerID="366fc18ce3b3f94145556568ce48d671a9ed8685e053edc15d84b2eb1b0ce90e" Oct 30 21:15:02.258967 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:02.258852 4926 generic.go:356] "Generic (PLEG): container finished" podID="e5160e9e-5873-438e-9679-7c0227fe753a" containerID="0d38cd0ecbbfe950db8270db48ef3c1b75941c13e43ba56f2cdfcf546e8cad09" exitCode=60 Oct 30 21:15:02.258967 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:02.258958 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5" event={"ID":"e5160e9e-5873-438e-9679-7c0227fe753a","Type":"ContainerDied","Data":"0d38cd0ecbbfe950db8270db48ef3c1b75941c13e43ba56f2cdfcf546e8cad09"} Oct 30 21:15:02.259214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:02.259194 4926 scope.go:122] "RemoveContainer" containerID="0d38cd0ecbbfe950db8270db48ef3c1b75941c13e43ba56f2cdfcf546e8cad09" Oct 30 21:15:02.261757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:02.261674 4926 generic.go:356] "Generic (PLEG): container finished" podID="81c872ad-0ad6-4f1a-93c4-fc79532d5e65" containerID="c04c476b090cd475148d149742d4894ac9b0f698ce56d2d225e3d6b057b79f6e" exitCode=60 Oct 30 21:15:02.261757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:02.261706 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c" event={"ID":"81c872ad-0ad6-4f1a-93c4-fc79532d5e65","Type":"ContainerDied","Data":"c04c476b090cd475148d149742d4894ac9b0f698ce56d2d225e3d6b057b79f6e"} Oct 30 21:15:02.261947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:02.261929 4926 scope.go:122] "RemoveContainer" containerID="c04c476b090cd475148d149742d4894ac9b0f698ce56d2d225e3d6b057b79f6e" Oct 30 21:15:02.263531 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:02.263510 4926 generic.go:356] "Generic (PLEG): container finished" podID="52b459ac-7beb-42a9-bcb6-9f193d76632c" containerID="8b08e9a270f21f80b2f75712614730aaf6261e5fa24f2e011db43e6f990e1f28" exitCode=60 Oct 30 21:15:02.263639 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:02.263590 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm" event={"ID":"52b459ac-7beb-42a9-bcb6-9f193d76632c","Type":"ContainerDied","Data":"8b08e9a270f21f80b2f75712614730aaf6261e5fa24f2e011db43e6f990e1f28"} Oct 30 21:15:02.263833 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:02.263819 4926 scope.go:122] "RemoveContainer" containerID="8b08e9a270f21f80b2f75712614730aaf6261e5fa24f2e011db43e6f990e1f28" Oct 30 21:15:02.265399 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:02.265377 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" event={"ID":"b42d62ed-b880-4a25-978c-47c703a5cdd3","Type":"ContainerStarted","Data":"eb143e0dcc55a237f94e1192b7449d8f1b34735672f2e91f4a70249e5d6cae62"} Oct 30 21:15:02.265470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:02.265424 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" event={"ID":"b42d62ed-b880-4a25-978c-47c703a5cdd3","Type":"ContainerStarted","Data":"b2c6af419e6768ff6f95627b06e6776daf8c5526183ecaaa2d1eb4e73f6b3055"} Oct 30 21:15:02.511003 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:02.510934 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" podStartSLOduration=2.5109114740000003 podStartE2EDuration="2.510911474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:02.509851374 +0000 UTC m=+889.630925133" watchObservedRunningTime="2026-10-30 21:15:02.510911474 +0000 UTC m=+889.631985222" Oct 30 21:15:02.995425 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:02.995350 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:03.001465 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:03.001144 4926 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:15:03.001465 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:03.001205 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" podUID="b42d62ed-b880-4a25-978c-47c703a5cdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 21:15:03.305100 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:03.304970 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2" event={"ID":"34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2","Type":"ContainerStarted","Data":"3eab87b4fab33f271fbb770f688078c1ea5359170382135e1d46ffc15f6eaeeb"} Oct 30 21:15:03.308152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:03.308125 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5" event={"ID":"e5160e9e-5873-438e-9679-7c0227fe753a","Type":"ContainerStarted","Data":"622ab09273ae17c457eab0a1ac9750b527bb2b803425c1c7578cc6fe7a83df12"} Oct 30 21:15:03.309918 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:03.309891 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c" event={"ID":"81c872ad-0ad6-4f1a-93c4-fc79532d5e65","Type":"ContainerStarted","Data":"bd62635c21a83d803449e55c761aecf4471f1b82e8c70aae98ccd17a4cf74463"} Oct 30 21:15:03.311837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:03.311812 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm" event={"ID":"52b459ac-7beb-42a9-bcb6-9f193d76632c","Type":"ContainerStarted","Data":"9ec46a3cedcdd00a6527feeee966fbbde81fc6a01c13cd1a75615a2551f131f2"} Oct 30 21:15:03.341822 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:03.341780 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2" podStartSLOduration=3.34176888 podStartE2EDuration="3.34176888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:03.341254549 +0000 UTC m=+890.462328295" watchObservedRunningTime="2026-10-30 21:15:03.34176888 +0000 UTC m=+890.462842661" Oct 30 21:15:03.440544 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:03.440480 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c" podStartSLOduration=3.440461096 podStartE2EDuration="3.440461096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:03.430839456 +0000 UTC m=+890.551913203" watchObservedRunningTime="2026-10-30 21:15:03.440461096 +0000 UTC m=+890.561534843" Oct 30 21:15:03.445350 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:03.445279 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm" podStartSLOduration=3.44526929 podStartE2EDuration="3.44526929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:03.378465683 +0000 UTC m=+890.499539429" watchObservedRunningTime="2026-10-30 21:15:03.44526929 +0000 UTC m=+890.566343036" Oct 30 21:15:03.490311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:03.486603 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:15:03.521675 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:03.521641 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:15:03.996454 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:03.996414 4926 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:15:03.996760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:03.996726 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" podUID="b42d62ed-b880-4a25-978c-47c703a5cdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 21:15:04.321395 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.321292 4926 generic.go:356] "Generic (PLEG): container finished" podID="e5160e9e-5873-438e-9679-7c0227fe753a" containerID="622ab09273ae17c457eab0a1ac9750b527bb2b803425c1c7578cc6fe7a83df12" exitCode=60 Oct 30 21:15:04.321395 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.321348 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5" event={"ID":"e5160e9e-5873-438e-9679-7c0227fe753a","Type":"ContainerDied","Data":"622ab09273ae17c457eab0a1ac9750b527bb2b803425c1c7578cc6fe7a83df12"} Oct 30 21:15:04.321850 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.321412 4926 scope.go:122] "RemoveContainer" containerID="0d38cd0ecbbfe950db8270db48ef3c1b75941c13e43ba56f2cdfcf546e8cad09" Oct 30 21:15:04.321850 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.321723 4926 scope.go:122] "RemoveContainer" containerID="622ab09273ae17c457eab0a1ac9750b527bb2b803425c1c7578cc6fe7a83df12" Oct 30 21:15:04.322286 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:15:04.321989 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5_odh-ai-gateway-infra(e5160e9e-5873-438e-9679-7c0227fe753a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5" podUID="e5160e9e-5873-438e-9679-7c0227fe753a" Oct 30 21:15:04.324829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.324807 4926 generic.go:356] "Generic (PLEG): container finished" podID="81c872ad-0ad6-4f1a-93c4-fc79532d5e65" containerID="bd62635c21a83d803449e55c761aecf4471f1b82e8c70aae98ccd17a4cf74463" exitCode=60 Oct 30 21:15:04.324933 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.324882 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c" event={"ID":"81c872ad-0ad6-4f1a-93c4-fc79532d5e65","Type":"ContainerDied","Data":"bd62635c21a83d803449e55c761aecf4471f1b82e8c70aae98ccd17a4cf74463"} Oct 30 21:15:04.325354 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.325321 4926 scope.go:122] "RemoveContainer" containerID="bd62635c21a83d803449e55c761aecf4471f1b82e8c70aae98ccd17a4cf74463" Oct 30 21:15:04.325666 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:15:04.325645 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c_odh-ai-gateway-infra(81c872ad-0ad6-4f1a-93c4-fc79532d5e65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c" podUID="81c872ad-0ad6-4f1a-93c4-fc79532d5e65" Oct 30 21:15:04.328270 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.328250 4926 generic.go:356] "Generic (PLEG): container finished" podID="52b459ac-7beb-42a9-bcb6-9f193d76632c" containerID="9ec46a3cedcdd00a6527feeee966fbbde81fc6a01c13cd1a75615a2551f131f2" exitCode=60 Oct 30 21:15:04.328369 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.328338 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm" event={"ID":"52b459ac-7beb-42a9-bcb6-9f193d76632c","Type":"ContainerDied","Data":"9ec46a3cedcdd00a6527feeee966fbbde81fc6a01c13cd1a75615a2551f131f2"} Oct 30 21:15:04.328609 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.328592 4926 scope.go:122] "RemoveContainer" containerID="9ec46a3cedcdd00a6527feeee966fbbde81fc6a01c13cd1a75615a2551f131f2" Oct 30 21:15:04.328911 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:15:04.328890 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm_odh-ai-gateway-infra(52b459ac-7beb-42a9-bcb6-9f193d76632c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm" podUID="52b459ac-7beb-42a9-bcb6-9f193d76632c" Oct 30 21:15:04.330363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.330320 4926 generic.go:356] "Generic (PLEG): container finished" podID="34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2" containerID="3eab87b4fab33f271fbb770f688078c1ea5359170382135e1d46ffc15f6eaeeb" exitCode=60 Oct 30 21:15:04.330458 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.330383 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2" event={"ID":"34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2","Type":"ContainerDied","Data":"3eab87b4fab33f271fbb770f688078c1ea5359170382135e1d46ffc15f6eaeeb"} Oct 30 21:15:04.330662 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.330638 4926 scope.go:122] "RemoveContainer" containerID="3eab87b4fab33f271fbb770f688078c1ea5359170382135e1d46ffc15f6eaeeb" Oct 30 21:15:04.330879 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:15:04.330860 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2_odh-ai-gateway-infra(34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2" podUID="34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2" Oct 30 21:15:04.337276 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.337256 4926 scope.go:122] "RemoveContainer" containerID="c04c476b090cd475148d149742d4894ac9b0f698ce56d2d225e3d6b057b79f6e" Oct 30 21:15:04.343073 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.343056 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:15:04.352682 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.352656 4926 scope.go:122] "RemoveContainer" containerID="8b08e9a270f21f80b2f75712614730aaf6261e5fa24f2e011db43e6f990e1f28" Oct 30 21:15:04.365938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.365918 4926 scope.go:122] "RemoveContainer" containerID="366fc18ce3b3f94145556568ce48d671a9ed8685e053edc15d84b2eb1b0ce90e" Oct 30 21:15:04.848046 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.847773 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6"] Oct 30 21:15:04.853368 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.853300 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6" Oct 30 21:15:04.929076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.922444 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:15:04.929076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.927941 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a61ce7-23f7-4719-8725-67972c9f6e88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9866b83-qwjf6\" (UID: \"84a61ce7-23f7-4719-8725-67972c9f6e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6" Oct 30 21:15:04.929076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.928072 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph9rc\" (UniqueName: \"kubernetes.io/projected/84a61ce7-23f7-4719-8725-67972c9f6e88-kube-api-access-ph9rc\") pod \"maas-api-revoke-sub-e9866b83-qwjf6\" (UID: \"84a61ce7-23f7-4719-8725-67972c9f6e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6" Oct 30 21:15:04.929076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.928113 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a61ce7-23f7-4719-8725-67972c9f6e88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9866b83-qwjf6\" (UID: \"84a61ce7-23f7-4719-8725-67972c9f6e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6" Oct 30 21:15:04.933700 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:04.930151 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6"] Oct 30 21:15:05.010057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:05.001435 4926 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:15:05.010057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:05.001506 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" podUID="b42d62ed-b880-4a25-978c-47c703a5cdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 21:15:05.029677 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:05.029651 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a61ce7-23f7-4719-8725-67972c9f6e88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9866b83-qwjf6\" (UID: \"84a61ce7-23f7-4719-8725-67972c9f6e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6" Oct 30 21:15:05.029914 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:05.029900 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph9rc\" (UniqueName: \"kubernetes.io/projected/84a61ce7-23f7-4719-8725-67972c9f6e88-kube-api-access-ph9rc\") pod \"maas-api-revoke-sub-e9866b83-qwjf6\" (UID: \"84a61ce7-23f7-4719-8725-67972c9f6e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6" Oct 30 21:15:05.029988 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:05.029980 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a61ce7-23f7-4719-8725-67972c9f6e88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9866b83-qwjf6\" (UID: \"84a61ce7-23f7-4719-8725-67972c9f6e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6" Oct 30 21:15:05.030906 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:05.030886 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a61ce7-23f7-4719-8725-67972c9f6e88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9866b83-qwjf6\" (UID: \"84a61ce7-23f7-4719-8725-67972c9f6e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6" Oct 30 21:15:05.032701 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:05.032685 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a61ce7-23f7-4719-8725-67972c9f6e88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9866b83-qwjf6\" (UID: \"84a61ce7-23f7-4719-8725-67972c9f6e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6" Oct 30 21:15:05.051746 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:05.051720 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph9rc\" (UniqueName: \"kubernetes.io/projected/84a61ce7-23f7-4719-8725-67972c9f6e88-kube-api-access-ph9rc\") pod \"maas-api-revoke-sub-e9866b83-qwjf6\" (UID: \"84a61ce7-23f7-4719-8725-67972c9f6e88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6" Oct 30 21:15:05.179052 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:05.178180 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6" Oct 30 21:15:05.352611 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:05.351762 4926 scope.go:122] "RemoveContainer" containerID="3eab87b4fab33f271fbb770f688078c1ea5359170382135e1d46ffc15f6eaeeb" Oct 30 21:15:05.352611 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:15:05.352125 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2_odh-ai-gateway-infra(34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2" podUID="34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2" Oct 30 21:15:05.355498 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:05.355387 4926 scope.go:122] "RemoveContainer" containerID="622ab09273ae17c457eab0a1ac9750b527bb2b803425c1c7578cc6fe7a83df12" Oct 30 21:15:05.356081 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:15:05.356054 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5_odh-ai-gateway-infra(e5160e9e-5873-438e-9679-7c0227fe753a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5" podUID="e5160e9e-5873-438e-9679-7c0227fe753a" Oct 30 21:15:05.358125 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:05.358103 4926 scope.go:122] "RemoveContainer" containerID="bd62635c21a83d803449e55c761aecf4471f1b82e8c70aae98ccd17a4cf74463" Oct 30 21:15:05.358550 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:15:05.358529 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c_odh-ai-gateway-infra(81c872ad-0ad6-4f1a-93c4-fc79532d5e65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c" podUID="81c872ad-0ad6-4f1a-93c4-fc79532d5e65" Oct 30 21:15:05.360104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:05.360086 4926 scope.go:122] "RemoveContainer" containerID="9ec46a3cedcdd00a6527feeee966fbbde81fc6a01c13cd1a75615a2551f131f2" Oct 30 21:15:05.360457 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:15:05.360424 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm_odh-ai-gateway-infra(52b459ac-7beb-42a9-bcb6-9f193d76632c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm" podUID="52b459ac-7beb-42a9-bcb6-9f193d76632c" Oct 30 21:15:05.529164 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:05.528885 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6"] Oct 30 21:15:05.999056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:05.995394 4926 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:15:05.999056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:05.995455 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" podUID="b42d62ed-b880-4a25-978c-47c703a5cdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 21:15:06.401964 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:06.401917 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6" event={"ID":"84a61ce7-23f7-4719-8725-67972c9f6e88","Type":"ContainerStarted","Data":"62f49b4607fb88b1375586d389d58e99121814e8d1e8e85799984cb0ccfffff5"} Oct 30 21:15:06.402719 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:06.402690 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6" event={"ID":"84a61ce7-23f7-4719-8725-67972c9f6e88","Type":"ContainerStarted","Data":"5487b1a24af5090b5c841e4214dfeab6412949f2240bc9179fe51e9d1c8bd336"} Oct 30 21:15:06.440711 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:06.440657 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6" podStartSLOduration=2.440643906 podStartE2EDuration="2.440643906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:06.439235936 +0000 UTC m=+893.560309681" watchObservedRunningTime="2026-10-30 21:15:06.440643906 +0000 UTC m=+893.561717652" Oct 30 21:15:06.995545 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:06.995450 4926 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:15:06.995789 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:06.995761 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" podUID="b42d62ed-b880-4a25-978c-47c703a5cdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 21:15:07.412966 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:07.412873 4926 generic.go:356] "Generic (PLEG): container finished" podID="84a61ce7-23f7-4719-8725-67972c9f6e88" containerID="62f49b4607fb88b1375586d389d58e99121814e8d1e8e85799984cb0ccfffff5" exitCode=0 Oct 30 21:15:07.413644 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:07.412968 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6" event={"ID":"84a61ce7-23f7-4719-8725-67972c9f6e88","Type":"ContainerDied","Data":"62f49b4607fb88b1375586d389d58e99121814e8d1e8e85799984cb0ccfffff5"} Oct 30 21:15:07.511435 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:15:07.511396 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:15:07.994064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:07.994016 4926 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 21:15:07.994252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:07.994090 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" podUID="b42d62ed-b880-4a25-978c-47c703a5cdd3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 21:15:08.606566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:08.606542 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6" Oct 30 21:15:08.692524 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:08.692443 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a61ce7-23f7-4719-8725-67972c9f6e88-kube-api-access-ph9rc\" (UniqueName: \"kubernetes.io/projected/84a61ce7-23f7-4719-8725-67972c9f6e88-kube-api-access-ph9rc\") pod \"84a61ce7-23f7-4719-8725-67972c9f6e88\" (UID: \"84a61ce7-23f7-4719-8725-67972c9f6e88\") " Oct 30 21:15:08.692767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:08.692741 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84a61ce7-23f7-4719-8725-67972c9f6e88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a61ce7-23f7-4719-8725-67972c9f6e88-maas-api-xks-ca\") pod \"84a61ce7-23f7-4719-8725-67972c9f6e88\" (UID: \"84a61ce7-23f7-4719-8725-67972c9f6e88\") " Oct 30 21:15:08.692988 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:08.692967 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a61ce7-23f7-4719-8725-67972c9f6e88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a61ce7-23f7-4719-8725-67972c9f6e88-maas-api-openshift-ca\") pod \"84a61ce7-23f7-4719-8725-67972c9f6e88\" (UID: \"84a61ce7-23f7-4719-8725-67972c9f6e88\") " Oct 30 21:15:08.693447 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:08.693419 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a61ce7-23f7-4719-8725-67972c9f6e88-maas-api-openshift-ca" pod "84a61ce7-23f7-4719-8725-67972c9f6e88" (UID: "84a61ce7-23f7-4719-8725-67972c9f6e88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:08.693946 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:08.693925 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a61ce7-23f7-4719-8725-67972c9f6e88-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:08.721093 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:08.720861 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a61ce7-23f7-4719-8725-67972c9f6e88-maas-api-xks-ca" pod "84a61ce7-23f7-4719-8725-67972c9f6e88" (UID: "84a61ce7-23f7-4719-8725-67972c9f6e88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:15:08.819056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:08.818981 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a61ce7-23f7-4719-8725-67972c9f6e88-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:08.863133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:08.836178 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a61ce7-23f7-4719-8725-67972c9f6e88-kube-api-access-ph9rc" pod "84a61ce7-23f7-4719-8725-67972c9f6e88" (UID: "84a61ce7-23f7-4719-8725-67972c9f6e88"). InnerVolumeSpecName "kube-api-access-ph9rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:08.926457 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:08.926419 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph9rc\" (UniqueName: \"kubernetes.io/projected/84a61ce7-23f7-4719-8725-67972c9f6e88-kube-api-access-ph9rc\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:09.191055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:09.188299 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:09.191055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:09.190853 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:09.192278 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:09.191925 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:15:09.605052 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:09.597468 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6" Oct 30 21:15:09.605052 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:09.601287 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6" event={"ID":"84a61ce7-23f7-4719-8725-67972c9f6e88","Type":"ContainerDied","Data":"5487b1a24af5090b5c841e4214dfeab6412949f2240bc9179fe51e9d1c8bd336"} Oct 30 21:15:09.605052 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:09.601340 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5487b1a24af5090b5c841e4214dfeab6412949f2240bc9179fe51e9d1c8bd336" Oct 30 21:15:10.666056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:10.665174 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:10.708101 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:10.694086 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:10.724057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:10.720942 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:10.745098 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:10.745061 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:10.785234 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:10.785193 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:10.808063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:10.807999 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:10.808937 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:10.808910 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:10.880067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:10.879942 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:11.332184 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:11.332130 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:11.419175 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:11.419130 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:11.469296 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:11.469251 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:11.747055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:11.730143 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:11.747055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:11.730881 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:15:11.756360 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:11.756320 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:11.767168 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:11.767139 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:15:11.836097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:11.804187 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:11.836097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:11.815157 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:11.836097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:11.815519 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:15:12.080643 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:12.080548 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:15:12.344282 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:12.344179 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:12.366638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:12.366610 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:15:12.498524 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:12.482265 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:12.498524 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:12.482647 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:15:12.593463 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:12.592129 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:12.603225 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:12.603139 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:12.636062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:12.624328 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 21:15:12.636062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:12.631974 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:12.903114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:12.876386 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 21:15:12.921878 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:12.921850 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:15:12.941340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:12.941157 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 21:15:12.946152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:12.941276 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 21:15:12.966060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:12.963962 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:15:13.036703 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:13.036655 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:15:13.036936 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:13.036797 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:15:13.207800 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:13.207770 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:15:13.208604 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:13.208585 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:15:14.008346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:14.008310 4926 scope.go:122] "RemoveContainer" containerID="622ab09273ae17c457eab0a1ac9750b527bb2b803425c1c7578cc6fe7a83df12" Oct 30 21:15:14.974237 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:14.974204 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5" event={"ID":"e5160e9e-5873-438e-9679-7c0227fe753a","Type":"ContainerStarted","Data":"3585a65f38d47bf7d4907ce7831d6b571afca77658632734234440a7c50e4376"} Oct 30 21:15:15.619787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.619755 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf"] Oct 30 21:15:15.620377 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.620362 4926 cpu_manager.go:414] "containerMap: removing container" podUID="84a61ce7-23f7-4719-8725-67972c9f6e88" containerName="revoke-keys" Oct 30 21:15:15.620422 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.620378 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a61ce7-23f7-4719-8725-67972c9f6e88" containerName="revoke-keys" Oct 30 21:15:15.620477 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.620468 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a61ce7-23f7-4719-8725-67972c9f6e88" containerName="revoke-keys" Oct 30 21:15:15.624206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.624186 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.652799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.652771 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf"] Oct 30 21:15:15.658188 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.658166 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-839163-openshift-default-dockercfg-lpmcf\"" Oct 30 21:15:15.711913 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.711870 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687"] Oct 30 21:15:15.716231 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.716211 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.733457 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.733426 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/472a982c-6acb-4164-96f6-5cf2904a4c8d-istiod-ca-cert\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.733585 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.733491 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-credential-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.733585 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.733537 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-podinfo\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.733585 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.733563 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-workload-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.733752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.733615 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-workload-certs\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.733752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.733640 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-envoy\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.733752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.733714 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-token\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.733913 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.733811 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhcwv\" (UniqueName: \"kubernetes.io/projected/472a982c-6acb-4164-96f6-5cf2904a4c8d-kube-api-access-mhcwv\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.733913 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.733843 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-data\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.751390 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.751366 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687"] Oct 30 21:15:15.834860 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.834828 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-data\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.835028 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.834870 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f607dfb4-2eab-4446-9501-1ac70b7129f7-istiod-ca-cert\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.835028 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.834894 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-workload-certs\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.835129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.835057 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-envoy\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.835129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.835101 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57c87\" (UniqueName: \"kubernetes.io/projected/f607dfb4-2eab-4446-9501-1ac70b7129f7-kube-api-access-57c87\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.835219 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.835135 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-podinfo\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.835257 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.835231 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/472a982c-6acb-4164-96f6-5cf2904a4c8d-istiod-ca-cert\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.835296 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.835248 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-data\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.835296 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.835270 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-data\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.835363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.835305 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-credential-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.835363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.835334 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-podinfo\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.835425 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.835376 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-workload-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.835462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.835427 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-workload-certs\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.835462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.835450 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-envoy\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.835540 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.835474 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-credential-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.835594 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.835546 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-token\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.835648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.835590 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-credential-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.835648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.835628 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-workload-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.835765 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.835658 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-token\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.835765 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.835685 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhcwv\" (UniqueName: \"kubernetes.io/projected/472a982c-6acb-4164-96f6-5cf2904a4c8d-kube-api-access-mhcwv\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.835765 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.835718 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-workload-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.835904 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.835773 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-workload-certs\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.835962 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.835941 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/472a982c-6acb-4164-96f6-5cf2904a4c8d-istiod-ca-cert\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.837680 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.837656 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-envoy\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.837884 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.837869 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-podinfo\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.856269 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.856249 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-token\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.858460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.858440 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhcwv\" (UniqueName: \"kubernetes.io/projected/472a982c-6acb-4164-96f6-5cf2904a4c8d-kube-api-access-mhcwv\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.936721 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.936691 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-workload-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.936876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.936729 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-token\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.936876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.936763 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f607dfb4-2eab-4446-9501-1ac70b7129f7-istiod-ca-cert\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.936876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.936784 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-workload-certs\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.936876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.936824 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-envoy\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.937121 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.936959 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57c87\" (UniqueName: \"kubernetes.io/projected/f607dfb4-2eab-4446-9501-1ac70b7129f7-kube-api-access-57c87\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.937121 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.937051 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-podinfo\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.937121 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.937117 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-data\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.937272 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.937139 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-workload-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.937272 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.937243 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-credential-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.937429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.937409 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-data\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.937556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.937536 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f607dfb4-2eab-4446-9501-1ac70b7129f7-istiod-ca-cert\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.937615 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.937566 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-credential-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.937652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.937609 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-workload-certs\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.937729 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.937709 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:15.939206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.939178 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-envoy\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.939745 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.939726 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-podinfo\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.958667 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.958647 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57c87\" (UniqueName: \"kubernetes.io/projected/f607dfb4-2eab-4446-9501-1ac70b7129f7-kube-api-access-57c87\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.958897 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.958877 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-token\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:15.990058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.988214 4926 generic.go:356] "Generic (PLEG): container finished" podID="e5160e9e-5873-438e-9679-7c0227fe753a" containerID="3585a65f38d47bf7d4907ce7831d6b571afca77658632734234440a7c50e4376" exitCode=60 Oct 30 21:15:15.990058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.988276 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5" event={"ID":"e5160e9e-5873-438e-9679-7c0227fe753a","Type":"ContainerDied","Data":"3585a65f38d47bf7d4907ce7831d6b571afca77658632734234440a7c50e4376"} Oct 30 21:15:15.990058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.988358 4926 scope.go:122] "RemoveContainer" containerID="622ab09273ae17c457eab0a1ac9750b527bb2b803425c1c7578cc6fe7a83df12" Oct 30 21:15:15.990058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:15.988745 4926 scope.go:122] "RemoveContainer" containerID="3585a65f38d47bf7d4907ce7831d6b571afca77658632734234440a7c50e4376" Oct 30 21:15:15.990058 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:15:15.989114 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5_odh-ai-gateway-infra(e5160e9e-5873-438e-9679-7c0227fe753a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5" podUID="e5160e9e-5873-438e-9679-7c0227fe753a" Oct 30 21:15:16.012347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.012324 4926 scope.go:122] "RemoveContainer" containerID="3eab87b4fab33f271fbb770f688078c1ea5359170382135e1d46ffc15f6eaeeb" Oct 30 21:15:16.027488 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.027320 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5"] Oct 30 21:15:16.031070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.030812 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:16.264934 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.264909 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf"] Oct 30 21:15:16.269245 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:15:16.269214 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod472a982c_6acb_4164_96f6_5cf2904a4c8d.slice/crio-598f267559a43ee780f8a61da595b61f91383af60b9b987af49788a2a5a92626 WatchSource:0}: Error finding container 598f267559a43ee780f8a61da595b61f91383af60b9b987af49788a2a5a92626: Status 404 returned error can't find the container with id 598f267559a43ee780f8a61da595b61f91383af60b9b987af49788a2a5a92626 Oct 30 21:15:16.272135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.272107 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:16.272233 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.272168 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:16.272233 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.272199 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:16.403075 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:15:16.403016 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf607dfb4_2eab_4446_9501_1ac70b7129f7.slice/crio-13213c52ed671612e5a8026c9935f456531c435dce0b5ad36470a460fba52a2d WatchSource:0}: Error finding container 13213c52ed671612e5a8026c9935f456531c435dce0b5ad36470a460fba52a2d: Status 404 returned error can't find the container with id 13213c52ed671612e5a8026c9935f456531c435dce0b5ad36470a460fba52a2d Oct 30 21:15:16.403223 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.403199 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687"] Oct 30 21:15:16.405932 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.405900 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:16.406064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.405963 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:16.406064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.405993 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:16.538311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.538226 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9"] Oct 30 21:15:16.546723 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.546697 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.570921 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.570895 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9"] Oct 30 21:15:16.603559 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.603515 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9"] Oct 30 21:15:16.647918 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.647887 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.654219 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.654192 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9"] Oct 30 21:15:16.661414 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.661356 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-workload-certs\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.661528 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.661436 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-podinfo\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.661528 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.661472 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdmkm\" (UniqueName: \"kubernetes.io/projected/8a80b31b-f4f7-47bc-abe7-27b0505919f5-kube-api-access-qdmkm\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.661652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.661553 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-data\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.661711 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.661653 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-workload-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.661711 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.661694 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-credential-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.661827 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.661718 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-envoy\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.661827 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.661762 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-token\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.661927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.661887 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istiod-ca-cert\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.762899 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.762871 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istiod-ca-cert\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.763209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.762943 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-data\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.763209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.763072 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdmkm\" (UniqueName: \"kubernetes.io/projected/8a80b31b-f4f7-47bc-abe7-27b0505919f5-kube-api-access-qdmkm\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.763209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.763143 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-credential-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.763209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.763197 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-workload-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.763614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.763232 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-token\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.763614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.763282 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrh52\" (UniqueName: \"kubernetes.io/projected/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-kube-api-access-rrh52\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.763614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.763339 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-workload-certs\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.763614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.763372 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-envoy\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.763614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.763407 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-podinfo\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.763614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.763442 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-token\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.763614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.763514 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-envoy\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.763614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.763543 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-workload-certs\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.763614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.763571 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istiod-ca-cert\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.764565 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.763712 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-data\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.764565 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.763777 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-workload-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.764565 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.763810 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-credential-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.764565 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.763852 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-podinfo\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.765020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.764992 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-data\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.765140 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.765118 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-workload-certs\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.765298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.765265 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-workload-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.765408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.765388 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istiod-ca-cert\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.765656 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.765634 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-credential-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.766880 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.766839 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-podinfo\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.771281 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.771250 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-envoy\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.795235 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.795114 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdmkm\" (UniqueName: \"kubernetes.io/projected/8a80b31b-f4f7-47bc-abe7-27b0505919f5-kube-api-access-qdmkm\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.795235 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.795166 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-token\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.864905 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.864867 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrh52\" (UniqueName: \"kubernetes.io/projected/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-kube-api-access-rrh52\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.864905 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.864907 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-workload-certs\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.865158 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.864944 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-podinfo\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.865158 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.864995 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-envoy\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.865271 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.865176 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istiod-ca-cert\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.865271 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.865248 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-data\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.865386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.865292 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-credential-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.865386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.865336 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-workload-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.865386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.865352 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-workload-certs\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.865386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.865368 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-token\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.865852 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.865827 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-credential-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.866026 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.866003 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istiod-ca-cert\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.866026 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.865880 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-workload-socket\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.866026 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.865883 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-data\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.867905 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.867844 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-envoy\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.868564 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.868539 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-podinfo\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.872467 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.872445 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:16.890769 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.890742 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-token\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.891428 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.891401 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrh52\" (UniqueName: \"kubernetes.io/projected/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-kube-api-access-rrh52\") pod \"e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:16.973675 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:16.972436 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:17.019206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.016616 4926 scope.go:122] "RemoveContainer" containerID="9ec46a3cedcdd00a6527feeee966fbbde81fc6a01c13cd1a75615a2551f131f2" Oct 30 21:15:17.019206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.018278 4926 scope.go:122] "RemoveContainer" containerID="bd62635c21a83d803449e55c761aecf4471f1b82e8c70aae98ccd17a4cf74463" Oct 30 21:15:17.157126 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.156719 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" event={"ID":"472a982c-6acb-4164-96f6-5cf2904a4c8d","Type":"ContainerStarted","Data":"5a9de4e8e86b80d8baaebb07cf93d204ba0ce046332869c8352c24fa936029fb"} Oct 30 21:15:17.157126 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.156795 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" event={"ID":"472a982c-6acb-4164-96f6-5cf2904a4c8d","Type":"ContainerStarted","Data":"598f267559a43ee780f8a61da595b61f91383af60b9b987af49788a2a5a92626"} Oct 30 21:15:17.165890 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.165747 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" event={"ID":"f607dfb4-2eab-4446-9501-1ac70b7129f7","Type":"ContainerStarted","Data":"c1e147327fbb07d55e1e2e0a9e411227c3db0a4e95991e004be57e2dad1b5e85"} Oct 30 21:15:17.165890 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.165809 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" event={"ID":"f607dfb4-2eab-4446-9501-1ac70b7129f7","Type":"ContainerStarted","Data":"13213c52ed671612e5a8026c9935f456531c435dce0b5ad36470a460fba52a2d"} Oct 30 21:15:17.314458 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.293733 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2" event={"ID":"34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2","Type":"ContainerStarted","Data":"5eba6788c988653467d4aa41e589cc1286d798dee581658ae7fdf2a6b6df5a65"} Oct 30 21:15:17.314458 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.294411 4926 scope.go:122] "RemoveContainer" containerID="5eba6788c988653467d4aa41e589cc1286d798dee581658ae7fdf2a6b6df5a65" Oct 30 21:15:17.314458 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:15:17.294982 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2_odh-ai-gateway-infra(34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2" podUID="34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2" Oct 30 21:15:17.348207 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.348116 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9"] Oct 30 21:15:17.349240 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.349204 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:17.349371 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.349335 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:17.349472 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.349430 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:17.350730 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.350677 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" podStartSLOduration=2.350658749 podStartE2EDuration="2.350658749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:17.298354245 +0000 UTC m=+904.419427995" watchObservedRunningTime="2026-10-30 21:15:17.350658749 +0000 UTC m=+904.471732496" Oct 30 21:15:17.486561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.465939 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a14ffe5-xtn68"] Oct 30 21:15:17.551803 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.551775 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a14ffe5-xtn68"] Oct 30 21:15:17.552060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.551999 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a14ffe5-xtn68" Oct 30 21:15:17.559420 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.559364 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" podStartSLOduration=2.559345775 podStartE2EDuration="2.559345775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:17.543612287 +0000 UTC m=+904.664686034" watchObservedRunningTime="2026-10-30 21:15:17.559345775 +0000 UTC m=+904.680419521" Oct 30 21:15:17.620277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.598272 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:15:17.664432 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.664398 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84rvp\" (UniqueName: \"kubernetes.io/projected/a780d05f-ec0c-4645-8388-07282de67878-kube-api-access-84rvp\") pod \"maas-api-revoke-sub-6a14ffe5-xtn68\" (UID: \"a780d05f-ec0c-4645-8388-07282de67878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a14ffe5-xtn68" Oct 30 21:15:17.665059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.664998 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a780d05f-ec0c-4645-8388-07282de67878-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a14ffe5-xtn68\" (UID: \"a780d05f-ec0c-4645-8388-07282de67878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a14ffe5-xtn68" Oct 30 21:15:17.665261 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.665245 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a780d05f-ec0c-4645-8388-07282de67878-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a14ffe5-xtn68\" (UID: \"a780d05f-ec0c-4645-8388-07282de67878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a14ffe5-xtn68" Oct 30 21:15:17.730427 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.730389 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9"] Oct 30 21:15:17.813142 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.774067 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84rvp\" (UniqueName: \"kubernetes.io/projected/a780d05f-ec0c-4645-8388-07282de67878-kube-api-access-84rvp\") pod \"maas-api-revoke-sub-6a14ffe5-xtn68\" (UID: \"a780d05f-ec0c-4645-8388-07282de67878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a14ffe5-xtn68" Oct 30 21:15:17.813542 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.813509 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a780d05f-ec0c-4645-8388-07282de67878-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a14ffe5-xtn68\" (UID: \"a780d05f-ec0c-4645-8388-07282de67878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a14ffe5-xtn68" Oct 30 21:15:17.816278 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.816250 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a780d05f-ec0c-4645-8388-07282de67878-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a14ffe5-xtn68\" (UID: \"a780d05f-ec0c-4645-8388-07282de67878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a14ffe5-xtn68" Oct 30 21:15:17.838608 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.838577 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a780d05f-ec0c-4645-8388-07282de67878-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a14ffe5-xtn68\" (UID: \"a780d05f-ec0c-4645-8388-07282de67878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a14ffe5-xtn68" Oct 30 21:15:17.843656 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.843597 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84rvp\" (UniqueName: \"kubernetes.io/projected/a780d05f-ec0c-4645-8388-07282de67878-kube-api-access-84rvp\") pod \"maas-api-revoke-sub-6a14ffe5-xtn68\" (UID: \"a780d05f-ec0c-4645-8388-07282de67878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a14ffe5-xtn68" Oct 30 21:15:17.845297 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.845258 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a780d05f-ec0c-4645-8388-07282de67878-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a14ffe5-xtn68\" (UID: \"a780d05f-ec0c-4645-8388-07282de67878\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a14ffe5-xtn68" Oct 30 21:15:17.847007 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.846976 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:17.847127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.847104 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:17.847791 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.847729 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:17.956209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.955971 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a14ffe5-xtn68" Oct 30 21:15:17.958392 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.957619 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:17.958392 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.957722 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 21:15:17.958392 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:17.957755 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" podUID="472a982c-6acb-4164-96f6-5cf2904a4c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 21:15:18.185296 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:18.173344 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:18.244410 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:18.243641 4926 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 21:15:18.244410 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:18.243702 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" podUID="f607dfb4-2eab-4446-9501-1ac70b7129f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 21:15:18.263792 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:18.263711 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5" Oct 30 21:15:18.410497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:18.409701 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5160e9e-5873-438e-9679-7c0227fe753a-kube-api-access-4zzl7\" (UniqueName: \"kubernetes.io/projected/e5160e9e-5873-438e-9679-7c0227fe753a-kube-api-access-4zzl7\") pod \"e5160e9e-5873-438e-9679-7c0227fe753a\" (UID: \"e5160e9e-5873-438e-9679-7c0227fe753a\") " Oct 30 21:15:18.411716 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:18.410894 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5160e9e-5873-438e-9679-7c0227fe753a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5160e9e-5873-438e-9679-7c0227fe753a-maas-api-service-ca\") pod \"e5160e9e-5873-438e-9679-7c0227fe753a\" (UID: \"e5160e9e-5873-438e-9679-7c0227fe753a\") " Oct 30 21:15:18.412061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:18.411997 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5160e9e-5873-438e-9679-7c0227fe753a-maas-api-service-ca" pod "e5160e9e-5873-438e-9679-7c0227fe753a" (UID: "e5160e9e-5873-438e-9679-7c0227fe753a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:18.527705 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:18.527319 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5160e9e-5873-438e-9679-7c0227fe753a-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:18.656061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:18.655835 4926 generic.go:356] "Generic (PLEG): container finished" podID="34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2" containerID="5eba6788c988653467d4aa41e589cc1286d798dee581658ae7fdf2a6b6df5a65" exitCode=60 Oct 30 21:15:18.656061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:18.655965 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2" event={"ID":"34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2","Type":"ContainerDied","Data":"5eba6788c988653467d4aa41e589cc1286d798dee581658ae7fdf2a6b6df5a65"} Oct 30 21:15:18.656061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:18.656022 4926 scope.go:122] "RemoveContainer" containerID="3eab87b4fab33f271fbb770f688078c1ea5359170382135e1d46ffc15f6eaeeb" Oct 30 21:15:18.657673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:18.656822 4926 scope.go:122] "RemoveContainer" containerID="5eba6788c988653467d4aa41e589cc1286d798dee581658ae7fdf2a6b6df5a65" Oct 30 21:15:18.657673 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:15:18.657159 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2_odh-ai-gateway-infra(34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2" podUID="34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2" Oct 30 21:15:18.664593 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:18.664407 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5160e9e-5873-438e-9679-7c0227fe753a-kube-api-access-4zzl7" pod "e5160e9e-5873-438e-9679-7c0227fe753a" (UID: "e5160e9e-5873-438e-9679-7c0227fe753a"). InnerVolumeSpecName "kube-api-access-4zzl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:18.715893 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:18.704678 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2"] Oct 30 21:15:18.761494 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:18.724825 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5" event={"ID":"e5160e9e-5873-438e-9679-7c0227fe753a","Type":"ContainerDied","Data":"a7b60accf6a048e76984747c8dbee69bd9b80475cae127b37095426ac8610f93"} Oct 30 21:15:18.761494 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:18.724956 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5" Oct 30 21:15:18.761494 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:18.750900 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zzl7\" (UniqueName: \"kubernetes.io/projected/e5160e9e-5873-438e-9679-7c0227fe753a-kube-api-access-4zzl7\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:18.884549 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:18.830890 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c" event={"ID":"81c872ad-0ad6-4f1a-93c4-fc79532d5e65","Type":"ContainerStarted","Data":"f3a8df2a800d033dd6401ad91776e5ea53ab84912be2f41fc6800b40c86d0c99"} Oct 30 21:15:18.884549 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:18.831418 4926 scope.go:122] "RemoveContainer" containerID="f3a8df2a800d033dd6401ad91776e5ea53ab84912be2f41fc6800b40c86d0c99" Oct 30 21:15:18.884549 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:15:18.831849 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c_odh-ai-gateway-infra(81c872ad-0ad6-4f1a-93c4-fc79532d5e65)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c" podUID="81c872ad-0ad6-4f1a-93c4-fc79532d5e65" Oct 30 21:15:18.943563 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:18.941441 4926 scope.go:122] "RemoveContainer" containerID="3585a65f38d47bf7d4907ce7831d6b571afca77658632734234440a7c50e4376" Oct 30 21:15:18.988125 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:18.987155 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 21:15:18.988125 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:18.987217 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" podUID="472a982c-6acb-4164-96f6-5cf2904a4c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 21:15:19.056055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:19.053551 4926 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 21:15:19.056055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:19.053603 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" podUID="f607dfb4-2eab-4446-9501-1ac70b7129f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 21:15:19.256239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:19.220878 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5"] Oct 30 21:15:19.256239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:19.220939 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm" event={"ID":"52b459ac-7beb-42a9-bcb6-9f193d76632c","Type":"ContainerStarted","Data":"bbe135ddb2347ecc28eef4cbba44e0e28dc3d9043f6a1b893d52c29e3f881387"} Oct 30 21:15:19.256239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:19.220992 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" event={"ID":"a3061a9a-e0c1-46c9-914b-a5aba158ec6a","Type":"ContainerStarted","Data":"1274ee38906004678694db8a766122bf44a4e6e33b277c07aa7ae47d50b01647"} Oct 30 21:15:19.256239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:19.221017 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" event={"ID":"8a80b31b-f4f7-47bc-abe7-27b0505919f5","Type":"ContainerStarted","Data":"7ef02ebb7fa0b590d8c6acd6c7ae552e30b1c1c8d62e7547cf1a0baf65de5de8"} Oct 30 21:15:19.256239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:19.221057 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" event={"ID":"8a80b31b-f4f7-47bc-abe7-27b0505919f5","Type":"ContainerStarted","Data":"a951f94531a5e5d193ee4fac3148ee7adb65f65bb19866bbc60ca1ae618261e2"} Oct 30 21:15:19.256239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:19.238997 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" podStartSLOduration=3.238977484 podStartE2EDuration="3.238977484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:19.156801299 +0000 UTC m=+906.277875049" watchObservedRunningTime="2026-10-30 21:15:19.238977484 +0000 UTC m=+906.360051231" Oct 30 21:15:19.256239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:19.241058 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889915-9sqk5"] Oct 30 21:15:19.256239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:19.248200 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" podStartSLOduration=3.248182022 podStartE2EDuration="3.248182022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:19.222347885 +0000 UTC m=+906.343421632" watchObservedRunningTime="2026-10-30 21:15:19.248182022 +0000 UTC m=+906.369255769" Oct 30 21:15:19.657400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:19.648447 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a14ffe5-xtn68"] Oct 30 21:15:19.693621 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:19.693577 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:19.789007 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:19.787563 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:19.938058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:19.914126 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:19.938058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:19.914246 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:15:19.938058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:19.914302 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" podUID="8a80b31b-f4f7-47bc-abe7-27b0505919f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:15:19.938965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:19.938779 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 21:15:19.938965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:19.938852 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" podUID="472a982c-6acb-4164-96f6-5cf2904a4c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 21:15:19.967981 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:19.967937 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:20.215634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.065581 4926 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 21:15:20.215979 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.215927 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" podUID="f607dfb4-2eab-4446-9501-1ac70b7129f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 21:15:20.216171 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.122555 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:20.217291 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.067100 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:20.217511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.136185 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:20.218065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.067174 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:15:20.333072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.330182 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" podUID="a3061a9a-e0c1-46c9-914b-a5aba158ec6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:15:20.333072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.225424 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:20.333072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.122794 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 21:15:20.450155 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.450098 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c"] Oct 30 21:15:20.530230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.521153 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:15:20.550111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.543579 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm"] Oct 30 21:15:20.550111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.543672 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" event={"ID":"a3061a9a-e0c1-46c9-914b-a5aba158ec6a","Type":"ContainerStarted","Data":"6f0b7410464cb92dda1adffffb7ccf1e766b0a966d29d90b04e90754fb97bb65"} Oct 30 21:15:20.604324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.583734 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a14ffe5-xtn68" event={"ID":"a780d05f-ec0c-4645-8388-07282de67878","Type":"ContainerStarted","Data":"122cd5ed28e1335bd79ade9552ac92cda6daade3547779ccaf7a326337575523"} Oct 30 21:15:20.643904 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.643855 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:20.655811 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.655771 4926 generic.go:356] "Generic (PLEG): container finished" podID="81c872ad-0ad6-4f1a-93c4-fc79532d5e65" containerID="f3a8df2a800d033dd6401ad91776e5ea53ab84912be2f41fc6800b40c86d0c99" exitCode=60 Oct 30 21:15:20.656075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.656053 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c" event={"ID":"81c872ad-0ad6-4f1a-93c4-fc79532d5e65","Type":"ContainerDied","Data":"f3a8df2a800d033dd6401ad91776e5ea53ab84912be2f41fc6800b40c86d0c99"} Oct 30 21:15:20.656250 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.656229 4926 scope.go:122] "RemoveContainer" containerID="bd62635c21a83d803449e55c761aecf4471f1b82e8c70aae98ccd17a4cf74463" Oct 30 21:15:20.707282 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.707228 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:20.708558 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.708537 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:15:20.740476 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.740441 4926 generic.go:356] "Generic (PLEG): container finished" podID="52b459ac-7beb-42a9-bcb6-9f193d76632c" containerID="bbe135ddb2347ecc28eef4cbba44e0e28dc3d9043f6a1b893d52c29e3f881387" exitCode=60 Oct 30 21:15:20.741824 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.741795 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm" event={"ID":"52b459ac-7beb-42a9-bcb6-9f193d76632c","Type":"ContainerDied","Data":"bbe135ddb2347ecc28eef4cbba44e0e28dc3d9043f6a1b893d52c29e3f881387"} Oct 30 21:15:20.790807 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.790711 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:20.812063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.791283 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:15:20.900115 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.839786 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:20.903685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.902858 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:15:20.903685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.902914 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" podUID="8a80b31b-f4f7-47bc-abe7-27b0505919f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:15:20.948054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.934505 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:20.987056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.969708 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:20.988242 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:20.988209 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:15:21.013160 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.013112 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" podUID="a3061a9a-e0c1-46c9-914b-a5aba158ec6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:15:21.053139 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.053052 4926 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 21:15:21.053398 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.053369 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" podUID="f607dfb4-2eab-4446-9501-1ac70b7129f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 21:15:21.061216 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.061185 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 21:15:21.062630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.062597 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" podUID="472a982c-6acb-4164-96f6-5cf2904a4c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 21:15:21.105171 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.092235 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:21.127646 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.127610 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5160e9e-5873-438e-9679-7c0227fe753a" path="/var/lib/kubelet/pods/e5160e9e-5873-438e-9679-7c0227fe753a/volumes" Oct 30 21:15:21.234225 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.234172 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 21:15:21.264870 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.238872 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:21.322147 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.245400 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:15:21.322147 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.267162 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:15:21.346717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.346673 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 21:15:21.348282 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.348256 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:15:21.675061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.670436 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 21:15:21.675061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.671180 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:15:21.675061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.672774 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:21.722845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.707897 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 21:15:21.727287 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.727246 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8k468" podUID="97d86e50-7f90-4650-bef4-2a4cd91c1d61" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.37:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:21.770696 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.770647 4926 scope.go:122] "RemoveContainer" containerID="9ec46a3cedcdd00a6527feeee966fbbde81fc6a01c13cd1a75615a2551f131f2" Oct 30 21:15:21.800952 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.792166 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:21.800952 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.793570 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:21.879637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.879593 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:21.880942 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.880919 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:15:21.891829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.891798 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:15:21.892087 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.892005 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" podUID="8a80b31b-f4f7-47bc-abe7-27b0505919f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:15:21.976729 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.971624 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 21:15:21.976729 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.973702 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:15:21.989929 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.989899 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 21:15:21.994282 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.994235 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" podUID="472a982c-6acb-4164-96f6-5cf2904a4c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 21:15:22.000296 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:21.974023 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:15:22.047158 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.047096 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:15:22.047339 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.047173 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" podUID="a3061a9a-e0c1-46c9-914b-a5aba158ec6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:15:22.087673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.087631 4926 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 21:15:22.087927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.087891 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" podUID="f607dfb4-2eab-4446-9501-1ac70b7129f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 21:15:22.088185 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.088161 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:22.102922 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.102885 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:22.104950 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.104926 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:15:22.454057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.109138 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:22.456338 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.261714 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:22.456715 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.261988 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:22.460103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.306610 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:22.486115 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.347126 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:22.521326 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.514071 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:22.576203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.563756 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:22.609555 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.609512 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded" Oct 30 21:15:22.625147 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.625104 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:22.727268 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.701590 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:22.727268 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.720629 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:22.818343 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.801047 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:15:22.888403 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.888356 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:22.952292 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.952246 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:22.974063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.973093 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 21:15:22.974063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.973150 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" podUID="472a982c-6acb-4164-96f6-5cf2904a4c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 21:15:22.974063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.973650 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:23.006197 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:22.980382 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:15:23.010075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.006934 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:15:23.030085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.020460 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:15:23.030085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.020514 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" podUID="a3061a9a-e0c1-46c9-914b-a5aba158ec6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:15:23.158079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.111280 4926 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 21:15:23.158079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.111341 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" podUID="f607dfb4-2eab-4446-9501-1ac70b7129f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 21:15:23.158079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.150365 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:15:23.158079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.150418 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" podUID="8a80b31b-f4f7-47bc-abe7-27b0505919f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:15:23.158079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.150967 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 21:15:23.469858 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.469790 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:23.474436 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.473121 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:23.488060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.487169 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:23.527071 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.491536 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:23.527071 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.491747 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 21:15:23.710941 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.709676 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 21:15:23.780360 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.778767 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2" Oct 30 21:15:23.790239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.790097 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm" Oct 30 21:15:23.862304 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.862254 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:23.864473 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.862991 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:23.866111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.866082 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:15:23.932715 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.881406 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c" Oct 30 21:15:23.954129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.905685 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:15:23.963151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.963085 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" podUID="8a80b31b-f4f7-47bc-abe7-27b0505919f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:15:23.963860 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.906408 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 21:15:23.979168 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.975316 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:23.979168 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.976501 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 21:15:23.979168 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.976542 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" podUID="472a982c-6acb-4164-96f6-5cf2904a4c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 21:15:23.997023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.996930 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2-maas-api-service-ca\") pod \"34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2\" (UID: \"34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2\") " Oct 30 21:15:24.015421 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.015298 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81c872ad-0ad6-4f1a-93c4-fc79532d5e65-kube-api-access-wtftb\" (UniqueName: \"kubernetes.io/projected/81c872ad-0ad6-4f1a-93c4-fc79532d5e65-kube-api-access-wtftb\") pod \"81c872ad-0ad6-4f1a-93c4-fc79532d5e65\" (UID: \"81c872ad-0ad6-4f1a-93c4-fc79532d5e65\") " Oct 30 21:15:24.021239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.017740 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52b459ac-7beb-42a9-bcb6-9f193d76632c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b459ac-7beb-42a9-bcb6-9f193d76632c-maas-api-service-ca\") pod \"52b459ac-7beb-42a9-bcb6-9f193d76632c\" (UID: \"52b459ac-7beb-42a9-bcb6-9f193d76632c\") " Oct 30 21:15:24.021239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.017815 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2-kube-api-access-9qgs6\" (UniqueName: \"kubernetes.io/projected/34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2-kube-api-access-9qgs6\") pod \"34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2\" (UID: \"34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2\") " Oct 30 21:15:24.021239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.017903 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81c872ad-0ad6-4f1a-93c4-fc79532d5e65-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81c872ad-0ad6-4f1a-93c4-fc79532d5e65-maas-api-service-ca\") pod \"81c872ad-0ad6-4f1a-93c4-fc79532d5e65\" (UID: \"81c872ad-0ad6-4f1a-93c4-fc79532d5e65\") " Oct 30 21:15:24.021239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.017942 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b459ac-7beb-42a9-bcb6-9f193d76632c-kube-api-access-lxc29\" (UniqueName: \"kubernetes.io/projected/52b459ac-7beb-42a9-bcb6-9f193d76632c-kube-api-access-lxc29\") pod \"52b459ac-7beb-42a9-bcb6-9f193d76632c\" (UID: \"52b459ac-7beb-42a9-bcb6-9f193d76632c\") " Oct 30 21:15:24.021239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.019478 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:24.021239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.020349 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2-maas-api-service-ca" pod "34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2" (UID: "34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:24.039093 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.023805 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:24.039343 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:23.999066 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 21:15:24.040813 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.040772 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b459ac-7beb-42a9-bcb6-9f193d76632c-maas-api-service-ca" pod "52b459ac-7beb-42a9-bcb6-9f193d76632c" (UID: "52b459ac-7beb-42a9-bcb6-9f193d76632c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:24.090059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.090002 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81c872ad-0ad6-4f1a-93c4-fc79532d5e65-maas-api-service-ca" pod "81c872ad-0ad6-4f1a-93c4-fc79532d5e65" (UID: "81c872ad-0ad6-4f1a-93c4-fc79532d5e65"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:24.115059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.098523 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:15:24.234058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.215640 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" podUID="a3061a9a-e0c1-46c9-914b-a5aba158ec6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:15:24.234058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.153995 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:24.234058 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:15:24.153788 4926 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.146s" Oct 30 21:15:24.234058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.216418 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:15:24.234058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.216474 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:15:24.234058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.216531 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:15:24.234058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.151946 4926 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 21:15:24.234058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.217883 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" podUID="f607dfb4-2eab-4446-9501-1ac70b7129f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 21:15:24.234058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.200210 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52b459ac-7beb-42a9-bcb6-9f193d76632c-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:24.234058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.217977 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81c872ad-0ad6-4f1a-93c4-fc79532d5e65-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:24.234058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.229431 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c" event={"ID":"81c872ad-0ad6-4f1a-93c4-fc79532d5e65","Type":"ContainerDied","Data":"9d5a1bee5a69196fd4a3cfa32e4e97abe8ba9b1159bfd08e103370cefeabc8ca"} Oct 30 21:15:24.234058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.229776 4926 scope.go:122] "RemoveContainer" containerID="f3a8df2a800d033dd6401ad91776e5ea53ab84912be2f41fc6800b40c86d0c99" Oct 30 21:15:24.257534 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.257436 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm" event={"ID":"52b459ac-7beb-42a9-bcb6-9f193d76632c","Type":"ContainerDied","Data":"8d4e956a970f7156c198a3f5c16e5d119a32909384f1142ac624d1abf3071fed"} Oct 30 21:15:24.297679 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.297642 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:15:24.297950 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.297922 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a14ffe5-xtn68" event={"ID":"a780d05f-ec0c-4645-8388-07282de67878","Type":"ContainerStarted","Data":"4b171cc43f13ce72af3b4daec1d55ffc03b96ec9951b79a326f2e00434deb2a0"} Oct 30 21:15:24.298146 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.298119 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2" event={"ID":"34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2","Type":"ContainerDied","Data":"3375a430f4a30dd9a6f3a4315014c02613c0217e884a062768ff40505031d06a"} Oct 30 21:15:24.313679 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.299540 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2" Oct 30 21:15:24.320737 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.314301 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm" Oct 30 21:15:24.413339 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.408328 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b459ac-7beb-42a9-bcb6-9f193d76632c-kube-api-access-lxc29" pod "52b459ac-7beb-42a9-bcb6-9f193d76632c" (UID: "52b459ac-7beb-42a9-bcb6-9f193d76632c"). InnerVolumeSpecName "kube-api-access-lxc29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:24.493398 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.474018 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:24.493398 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.474195 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 21:15:24.500058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.498262 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:24.500058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.498647 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:24.500058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.499022 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:15:24.630077 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.626101 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxc29\" (UniqueName: \"kubernetes.io/projected/52b459ac-7beb-42a9-bcb6-9f193d76632c-kube-api-access-lxc29\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:24.643221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.643173 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2-kube-api-access-9qgs6" pod "34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2" (UID: "34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2"). InnerVolumeSpecName "kube-api-access-9qgs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:24.645184 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.645146 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81c872ad-0ad6-4f1a-93c4-fc79532d5e65-kube-api-access-wtftb" pod "81c872ad-0ad6-4f1a-93c4-fc79532d5e65" (UID: "81c872ad-0ad6-4f1a-93c4-fc79532d5e65"). InnerVolumeSpecName "kube-api-access-wtftb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:24.650494 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.650475 4926 scope.go:122] "RemoveContainer" containerID="bbe135ddb2347ecc28eef4cbba44e0e28dc3d9043f6a1b893d52c29e3f881387" Oct 30 21:15:24.712138 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.712089 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:24.879941 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.879853 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qgs6\" (UniqueName: \"kubernetes.io/projected/34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2-kube-api-access-9qgs6\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:24.880491 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.880469 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtftb\" (UniqueName: \"kubernetes.io/projected/81c872ad-0ad6-4f1a-93c4-fc79532d5e65-kube-api-access-wtftb\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:24.929188 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.929150 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:24.941475 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.935186 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:24.941475 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.935792 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:15:24.946282 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.942695 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:24.970450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.959115 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:15:24.970450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.959178 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" podUID="8a80b31b-f4f7-47bc-abe7-27b0505919f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:15:24.970450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.961073 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 21:15:24.970450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.961116 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" podUID="472a982c-6acb-4164-96f6-5cf2904a4c8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 21:15:24.971430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.971046 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded" Oct 30 21:15:24.971430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.971138 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 21:15:25.014118 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:24.983599 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:25.030993 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.029130 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:15:25.055241 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.054172 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:25.112689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.105207 4926 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Oct 30 21:15:25.112689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.105278 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" podUID="f607dfb4-2eab-4446-9501-1ac70b7129f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Oct 30 21:15:25.112689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.107875 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:15:25.112689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.107919 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" podUID="a3061a9a-e0c1-46c9-914b-a5aba158ec6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:15:25.264291 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.256360 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:15:25.279091 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.258098 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:15:25.279383 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.279364 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:15:25.285116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.285088 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf"] Oct 30 21:15:25.335913 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.335872 4926 cpu_manager.go:414] "containerMap: removing container" podUID="e5160e9e-5873-438e-9679-7c0227fe753a" containerName="cleanup" Oct 30 21:15:25.336173 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.336153 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5160e9e-5873-438e-9679-7c0227fe753a" containerName="cleanup" Oct 30 21:15:25.336302 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.336290 4926 cpu_manager.go:414] "containerMap: removing container" podUID="81c872ad-0ad6-4f1a-93c4-fc79532d5e65" containerName="cleanup" Oct 30 21:15:25.336401 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.336389 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81c872ad-0ad6-4f1a-93c4-fc79532d5e65" containerName="cleanup" Oct 30 21:15:25.336495 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.336485 4926 cpu_manager.go:414] "containerMap: removing container" podUID="e5160e9e-5873-438e-9679-7c0227fe753a" containerName="cleanup" Oct 30 21:15:25.336595 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.336584 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5160e9e-5873-438e-9679-7c0227fe753a" containerName="cleanup" Oct 30 21:15:25.336700 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.336689 4926 cpu_manager.go:414] "containerMap: removing container" podUID="52b459ac-7beb-42a9-bcb6-9f193d76632c" containerName="cleanup" Oct 30 21:15:25.336805 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.336792 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b459ac-7beb-42a9-bcb6-9f193d76632c" containerName="cleanup" Oct 30 21:15:25.336914 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.336903 4926 cpu_manager.go:414] "containerMap: removing container" podUID="34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2" containerName="cleanup" Oct 30 21:15:25.337008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.336997 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2" containerName="cleanup" Oct 30 21:15:25.337131 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.337120 4926 cpu_manager.go:414] "containerMap: removing container" podUID="81c872ad-0ad6-4f1a-93c4-fc79532d5e65" containerName="cleanup" Oct 30 21:15:25.337230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.337218 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81c872ad-0ad6-4f1a-93c4-fc79532d5e65" containerName="cleanup" Oct 30 21:15:25.337346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.337335 4926 cpu_manager.go:414] "containerMap: removing container" podUID="34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2" containerName="cleanup" Oct 30 21:15:25.337440 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.337429 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2" containerName="cleanup" Oct 30 21:15:25.337561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.337550 4926 cpu_manager.go:414] "containerMap: removing container" podUID="52b459ac-7beb-42a9-bcb6-9f193d76632c" containerName="cleanup" Oct 30 21:15:25.337655 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.337644 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b459ac-7beb-42a9-bcb6-9f193d76632c" containerName="cleanup" Oct 30 21:15:25.337925 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.337911 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="81c872ad-0ad6-4f1a-93c4-fc79532d5e65" containerName="cleanup" Oct 30 21:15:25.338026 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.338016 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5160e9e-5873-438e-9679-7c0227fe753a" containerName="cleanup" Oct 30 21:15:25.338204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.338191 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="81c872ad-0ad6-4f1a-93c4-fc79532d5e65" containerName="cleanup" Oct 30 21:15:25.338293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.338283 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2" containerName="cleanup" Oct 30 21:15:25.338374 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.338364 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2" containerName="cleanup" Oct 30 21:15:25.338446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.338437 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b459ac-7beb-42a9-bcb6-9f193d76632c" containerName="cleanup" Oct 30 21:15:25.338524 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.338515 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b459ac-7beb-42a9-bcb6-9f193d76632c" containerName="cleanup" Oct 30 21:15:25.338620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.338609 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5160e9e-5873-438e-9679-7c0227fe753a" containerName="cleanup" Oct 30 21:15:25.449809 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.449773 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:15:25.450176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.450159 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf"] Oct 30 21:15:25.482099 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.450096 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf" Oct 30 21:15:25.561425 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.550312 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 21:15:25.588363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.561703 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:25.664057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.663997 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqsqf\" (UniqueName: \"kubernetes.io/projected/d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4-kube-api-access-dqsqf\") pod \"maas-api-revoke-sub-0556107e-j7shf\" (UID: \"d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf" Oct 30 21:15:25.664371 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.664347 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0556107e-j7shf\" (UID: \"d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf" Oct 30 21:15:25.664620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.664600 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0556107e-j7shf\" (UID: \"d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf" Oct 30 21:15:25.695058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.691138 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:15:25.717166 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.702120 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:15:25.758155 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.721237 4926 scope.go:122] "RemoveContainer" containerID="5eba6788c988653467d4aa41e589cc1286d798dee581658ae7fdf2a6b6df5a65" Oct 30 21:15:25.947095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.918858 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:15:25.947095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.918908 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" podUID="8a80b31b-f4f7-47bc-abe7-27b0505919f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:15:25.947095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.920083 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0556107e-j7shf\" (UID: \"d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf" Oct 30 21:15:25.947095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.920253 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0556107e-j7shf\" (UID: \"d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf" Oct 30 21:15:25.947095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.920536 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqsqf\" (UniqueName: \"kubernetes.io/projected/d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4-kube-api-access-dqsqf\") pod \"maas-api-revoke-sub-0556107e-j7shf\" (UID: \"d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf" Oct 30 21:15:25.973487 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.949349 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0556107e-j7shf\" (UID: \"d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf" Oct 30 21:15:25.973487 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:25.949883 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c" Oct 30 21:15:26.008802 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.008672 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:15:26.035408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.035366 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:26.045219 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.045109 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 21:15:26.047638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.046361 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:15:26.047638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.046405 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" podUID="a3061a9a-e0c1-46c9-914b-a5aba158ec6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:15:26.084063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.059427 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 21:15:26.111637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.085226 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 21:15:26.116048 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.112765 4926 generic.go:356] "Generic (PLEG): container finished" podID="a780d05f-ec0c-4645-8388-07282de67878" containerID="4b171cc43f13ce72af3b4daec1d55ffc03b96ec9951b79a326f2e00434deb2a0" exitCode=0 Oct 30 21:15:26.200485 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.124847 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:26.201884 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.124887 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:26.206103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.125720 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a14ffe5-xtn68" event={"ID":"a780d05f-ec0c-4645-8388-07282de67878","Type":"ContainerDied","Data":"4b171cc43f13ce72af3b4daec1d55ffc03b96ec9951b79a326f2e00434deb2a0"} Oct 30 21:15:26.206294 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.157900 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:26.207849 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.199523 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0556107e-j7shf\" (UID: \"d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf" Oct 30 21:15:26.283423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.280159 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:26.296237 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.296122 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 21:15:26.300064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.297014 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:15:26.342062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.309605 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:26.532214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.530389 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm"] Oct 30 21:15:26.572235 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.572209 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889915-rjshm"] Oct 30 21:15:26.574610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.574590 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:15:26.604063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.593608 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:26.604063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.593663 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:15:26.604063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.593713 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:26.604063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.593751 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:15:26.604063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.593785 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:15:26.604063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.575584 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqsqf\" (UniqueName: \"kubernetes.io/projected/d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4-kube-api-access-dqsqf\") pod \"maas-api-revoke-sub-0556107e-j7shf\" (UID: \"d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf" Oct 30 21:15:26.604063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.603853 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:26.604063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.603896 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:26.627443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.627415 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:15:26.630172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.630148 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:15:26.646847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.646821 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:15:26.651312 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.651292 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf" Oct 30 21:15:26.652452 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.652435 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:15:26.722415 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.722384 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded" Oct 30 21:15:26.800135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.783139 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 21:15:26.800135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.783965 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 21:15:26.800135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.784832 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:15:26.950670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.950640 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:15:26.951277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.951257 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:15:26.971413 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.962775 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Oct 30 21:15:26.971413 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.962835 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" podUID="8a80b31b-f4f7-47bc-abe7-27b0505919f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Oct 30 21:15:26.971413 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.963378 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:15:26.975756 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.975731 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:15:26.975943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:26.975920 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" podUID="a3061a9a-e0c1-46c9-914b-a5aba158ec6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:15:27.061874 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:27.061768 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52b459ac-7beb-42a9-bcb6-9f193d76632c" path="/var/lib/kubelet/pods/52b459ac-7beb-42a9-bcb6-9f193d76632c/volumes" Oct 30 21:15:27.062348 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:27.062313 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c"] Oct 30 21:15:27.062526 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:27.062509 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889915-d598c"] Oct 30 21:15:27.065577 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:27.065538 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a14ffe5-xtn68" Oct 30 21:15:27.182252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:27.175591 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a780d05f-ec0c-4645-8388-07282de67878-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a780d05f-ec0c-4645-8388-07282de67878-maas-api-xks-ca\") pod \"a780d05f-ec0c-4645-8388-07282de67878\" (UID: \"a780d05f-ec0c-4645-8388-07282de67878\") " Oct 30 21:15:27.182252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:27.175713 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a780d05f-ec0c-4645-8388-07282de67878-kube-api-access-84rvp\" (UniqueName: \"kubernetes.io/projected/a780d05f-ec0c-4645-8388-07282de67878-kube-api-access-84rvp\") pod \"a780d05f-ec0c-4645-8388-07282de67878\" (UID: \"a780d05f-ec0c-4645-8388-07282de67878\") " Oct 30 21:15:27.182252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:27.175856 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a780d05f-ec0c-4645-8388-07282de67878-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a780d05f-ec0c-4645-8388-07282de67878-maas-api-openshift-ca\") pod \"a780d05f-ec0c-4645-8388-07282de67878\" (UID: \"a780d05f-ec0c-4645-8388-07282de67878\") " Oct 30 21:15:27.223856 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:27.222444 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a780d05f-ec0c-4645-8388-07282de67878-maas-api-xks-ca" pod "a780d05f-ec0c-4645-8388-07282de67878" (UID: "a780d05f-ec0c-4645-8388-07282de67878"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:15:27.224640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:27.224599 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a780d05f-ec0c-4645-8388-07282de67878-kube-api-access-84rvp" pod "a780d05f-ec0c-4645-8388-07282de67878" (UID: "a780d05f-ec0c-4645-8388-07282de67878"). InnerVolumeSpecName "kube-api-access-84rvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:27.240543 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:27.240509 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2"] Oct 30 21:15:27.250001 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:27.248565 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889915-fk9q2"] Oct 30 21:15:27.253544 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:27.253523 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a14ffe5-xtn68" Oct 30 21:15:27.253813 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:27.253624 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a14ffe5-xtn68" event={"ID":"a780d05f-ec0c-4645-8388-07282de67878","Type":"ContainerDied","Data":"122cd5ed28e1335bd79ade9552ac92cda6daade3547779ccaf7a326337575523"} Oct 30 21:15:27.253991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:27.253970 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="122cd5ed28e1335bd79ade9552ac92cda6daade3547779ccaf7a326337575523" Oct 30 21:15:27.255078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:27.255011 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a780d05f-ec0c-4645-8388-07282de67878-maas-api-openshift-ca" pod "a780d05f-ec0c-4645-8388-07282de67878" (UID: "a780d05f-ec0c-4645-8388-07282de67878"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:27.315450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:27.311882 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84rvp\" (UniqueName: \"kubernetes.io/projected/a780d05f-ec0c-4645-8388-07282de67878-kube-api-access-84rvp\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:27.315450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:27.311916 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a780d05f-ec0c-4645-8388-07282de67878-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:27.315450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:27.311932 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a780d05f-ec0c-4645-8388-07282de67878-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:27.433315 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:27.432947 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf"] Oct 30 21:15:27.435252 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:15:27.434232 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd70aee08_a8a7_4ef5_95b6_1ea7f0f0f1e4.slice/crio-e332d82b888b3469835ce1e64eda68b8f38c52d0023788c7f008b4344f17e0ed WatchSource:0}: Error finding container e332d82b888b3469835ce1e64eda68b8f38c52d0023788c7f008b4344f17e0ed: Status 404 returned error can't find the container with id e332d82b888b3469835ce1e64eda68b8f38c52d0023788c7f008b4344f17e0ed Oct 30 21:15:27.973980 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:27.973882 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Oct 30 21:15:27.973980 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:27.973950 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" podUID="a3061a9a-e0c1-46c9-914b-a5aba158ec6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Oct 30 21:15:28.139465 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:28.139432 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:28.140300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:28.140267 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:28.142229 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:28.142202 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:15:28.297796 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:28.297711 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf" event={"ID":"d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4","Type":"ContainerStarted","Data":"6c2e2fd2b93646b708df51b920e5bbfd662548cc01c72b4adf667806aa726aa7"} Oct 30 21:15:28.297972 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:28.297841 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf" event={"ID":"d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4","Type":"ContainerStarted","Data":"e332d82b888b3469835ce1e64eda68b8f38c52d0023788c7f008b4344f17e0ed"} Oct 30 21:15:28.340444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:28.340388 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf" podStartSLOduration=5.340370092 podStartE2EDuration="5.340370092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:28.333164632 +0000 UTC m=+915.454238379" watchObservedRunningTime="2026-10-30 21:15:28.340370092 +0000 UTC m=+915.461443838" Oct 30 21:15:29.017865 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:29.017826 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2" path="/var/lib/kubelet/pods/34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2/volumes" Oct 30 21:15:29.018555 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:29.018531 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81c872ad-0ad6-4f1a-93c4-fc79532d5e65" path="/var/lib/kubelet/pods/81c872ad-0ad6-4f1a-93c4-fc79532d5e65/volumes" Oct 30 21:15:29.022980 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:29.022955 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:29.023792 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:29.023769 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:29.025017 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:29.024997 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:15:29.306278 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:29.306181 4926 generic.go:356] "Generic (PLEG): container finished" podID="d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4" containerID="6c2e2fd2b93646b708df51b920e5bbfd662548cc01c72b4adf667806aa726aa7" exitCode=0 Oct 30 21:15:29.306452 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:29.306270 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf" event={"ID":"d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4","Type":"ContainerDied","Data":"6c2e2fd2b93646b708df51b920e5bbfd662548cc01c72b4adf667806aa726aa7"} Oct 30 21:15:30.476969 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:30.476943 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf" Oct 30 21:15:30.561116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:30.561072 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4-maas-api-openshift-ca\") pod \"d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4\" (UID: \"d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4\") " Oct 30 21:15:30.561420 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:30.561385 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4-kube-api-access-dqsqf\" (UniqueName: \"kubernetes.io/projected/d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4-kube-api-access-dqsqf\") pod \"d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4\" (UID: \"d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4\") " Oct 30 21:15:30.561623 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:30.561423 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4-maas-api-xks-ca\") pod \"d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4\" (UID: \"d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4\") " Oct 30 21:15:30.561623 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:30.561459 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4-maas-api-openshift-ca" pod "d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4" (UID: "d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:30.562080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:30.562046 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:30.564136 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:30.564108 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4-kube-api-access-dqsqf" pod "d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4" (UID: "d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4"). InnerVolumeSpecName "kube-api-access-dqsqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:30.564254 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:30.564222 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4-maas-api-xks-ca" pod "d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4" (UID: "d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:15:30.663742 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:30.663703 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqsqf\" (UniqueName: \"kubernetes.io/projected/d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4-kube-api-access-dqsqf\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:30.663742 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:30.663742 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:31.324821 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:31.324738 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf" Oct 30 21:15:31.324821 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:31.324753 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf" event={"ID":"d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4","Type":"ContainerDied","Data":"e332d82b888b3469835ce1e64eda68b8f38c52d0023788c7f008b4344f17e0ed"} Oct 30 21:15:31.325003 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:31.324840 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e332d82b888b3469835ce1e64eda68b8f38c52d0023788c7f008b4344f17e0ed" Oct 30 21:15:37.486377 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:15:37.486331 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:15:37.728294 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.728261 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f6fad1-n7pln"] Oct 30 21:15:37.728876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.728862 4926 cpu_manager.go:414] "containerMap: removing container" podUID="81c872ad-0ad6-4f1a-93c4-fc79532d5e65" containerName="cleanup" Oct 30 21:15:37.728930 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.728877 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81c872ad-0ad6-4f1a-93c4-fc79532d5e65" containerName="cleanup" Oct 30 21:15:37.728930 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.728900 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4" containerName="revoke-keys" Oct 30 21:15:37.728930 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.728905 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4" containerName="revoke-keys" Oct 30 21:15:37.729021 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.728931 4926 cpu_manager.go:414] "containerMap: removing container" podUID="a780d05f-ec0c-4645-8388-07282de67878" containerName="revoke-keys" Oct 30 21:15:37.729021 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.728936 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a780d05f-ec0c-4645-8388-07282de67878" containerName="revoke-keys" Oct 30 21:15:37.729021 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.728944 4926 cpu_manager.go:414] "containerMap: removing container" podUID="e5160e9e-5873-438e-9679-7c0227fe753a" containerName="cleanup" Oct 30 21:15:37.729021 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.728949 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5160e9e-5873-438e-9679-7c0227fe753a" containerName="cleanup" Oct 30 21:15:37.729021 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.728957 4926 cpu_manager.go:414] "containerMap: removing container" podUID="52b459ac-7beb-42a9-bcb6-9f193d76632c" containerName="cleanup" Oct 30 21:15:37.729021 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.728961 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b459ac-7beb-42a9-bcb6-9f193d76632c" containerName="cleanup" Oct 30 21:15:37.729021 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.728971 4926 cpu_manager.go:414] "containerMap: removing container" podUID="34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2" containerName="cleanup" Oct 30 21:15:37.729021 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.728975 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2" containerName="cleanup" Oct 30 21:15:37.729314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.729081 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b459ac-7beb-42a9-bcb6-9f193d76632c" containerName="cleanup" Oct 30 21:15:37.729314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.729095 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="a780d05f-ec0c-4645-8388-07282de67878" containerName="revoke-keys" Oct 30 21:15:37.729314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.729105 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5160e9e-5873-438e-9679-7c0227fe753a" containerName="cleanup" Oct 30 21:15:37.729314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.729113 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="81c872ad-0ad6-4f1a-93c4-fc79532d5e65" containerName="cleanup" Oct 30 21:15:37.729314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.729119 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4" containerName="revoke-keys" Oct 30 21:15:37.729314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.729125 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f1501c-70e6-4e82-aa6c-ac0e0bbfb0d2" containerName="cleanup" Oct 30 21:15:37.734002 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.733983 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f6fad1-n7pln" Oct 30 21:15:37.749612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.749583 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f6fad1-n7pln"] Oct 30 21:15:37.754342 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.754324 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:15:37.754482 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.754466 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:15:37.764986 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.764961 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrhtk\" (UniqueName: \"kubernetes.io/projected/c1708326-0038-4cdb-8f8a-5e2577634a66-kube-api-access-rrhtk\") pod \"maas-api-revoke-sub-89f6fad1-n7pln\" (UID: \"c1708326-0038-4cdb-8f8a-5e2577634a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f6fad1-n7pln" Oct 30 21:15:37.765153 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.765134 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1708326-0038-4cdb-8f8a-5e2577634a66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89f6fad1-n7pln\" (UID: \"c1708326-0038-4cdb-8f8a-5e2577634a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f6fad1-n7pln" Oct 30 21:15:37.765219 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.765163 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1708326-0038-4cdb-8f8a-5e2577634a66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89f6fad1-n7pln\" (UID: \"c1708326-0038-4cdb-8f8a-5e2577634a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f6fad1-n7pln" Oct 30 21:15:37.866598 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.866564 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrhtk\" (UniqueName: \"kubernetes.io/projected/c1708326-0038-4cdb-8f8a-5e2577634a66-kube-api-access-rrhtk\") pod \"maas-api-revoke-sub-89f6fad1-n7pln\" (UID: \"c1708326-0038-4cdb-8f8a-5e2577634a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f6fad1-n7pln" Oct 30 21:15:37.866782 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.866687 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1708326-0038-4cdb-8f8a-5e2577634a66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89f6fad1-n7pln\" (UID: \"c1708326-0038-4cdb-8f8a-5e2577634a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f6fad1-n7pln" Oct 30 21:15:37.866782 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.866716 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1708326-0038-4cdb-8f8a-5e2577634a66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89f6fad1-n7pln\" (UID: \"c1708326-0038-4cdb-8f8a-5e2577634a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f6fad1-n7pln" Oct 30 21:15:37.867494 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.867471 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1708326-0038-4cdb-8f8a-5e2577634a66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89f6fad1-n7pln\" (UID: \"c1708326-0038-4cdb-8f8a-5e2577634a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f6fad1-n7pln" Oct 30 21:15:37.869162 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.869137 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1708326-0038-4cdb-8f8a-5e2577634a66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89f6fad1-n7pln\" (UID: \"c1708326-0038-4cdb-8f8a-5e2577634a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f6fad1-n7pln" Oct 30 21:15:37.885917 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:37.885888 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrhtk\" (UniqueName: \"kubernetes.io/projected/c1708326-0038-4cdb-8f8a-5e2577634a66-kube-api-access-rrhtk\") pod \"maas-api-revoke-sub-89f6fad1-n7pln\" (UID: \"c1708326-0038-4cdb-8f8a-5e2577634a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f6fad1-n7pln" Oct 30 21:15:38.051627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:38.051553 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f6fad1-n7pln" Oct 30 21:15:38.212096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:38.212059 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:15:38.355773 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:38.355745 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f6fad1-n7pln"] Oct 30 21:15:38.355902 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:15:38.355831 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1708326_0038_4cdb_8f8a_5e2577634a66.slice/crio-549ee8138f5ae7c5c4013ea05329b0c68b914060acfd5d41fd55aef9c2452aaf WatchSource:0}: Error finding container 549ee8138f5ae7c5c4013ea05329b0c68b914060acfd5d41fd55aef9c2452aaf: Status 404 returned error can't find the container with id 549ee8138f5ae7c5c4013ea05329b0c68b914060acfd5d41fd55aef9c2452aaf Oct 30 21:15:38.383664 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:38.383643 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f6fad1-n7pln" event={"ID":"c1708326-0038-4cdb-8f8a-5e2577634a66","Type":"ContainerStarted","Data":"549ee8138f5ae7c5c4013ea05329b0c68b914060acfd5d41fd55aef9c2452aaf"} Oct 30 21:15:39.397687 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:39.397651 4926 generic.go:356] "Generic (PLEG): container finished" podID="c1708326-0038-4cdb-8f8a-5e2577634a66" containerID="1812dcf0995ac4f1ae2070dc062c48f1a02ca141afbc2f350b7e5262f0dd0e33" exitCode=0 Oct 30 21:15:39.398140 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:39.397741 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f6fad1-n7pln" event={"ID":"c1708326-0038-4cdb-8f8a-5e2577634a66","Type":"ContainerDied","Data":"1812dcf0995ac4f1ae2070dc062c48f1a02ca141afbc2f350b7e5262f0dd0e33"} Oct 30 21:15:40.897131 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:40.896135 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:40.956057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:40.955137 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:40.976058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:40.973154 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 21:15:41.004057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:41.001354 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:41.075056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:41.074194 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:41.641097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:41.520237 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 21:15:41.723112 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:41.716135 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:41.806066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:41.796275 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:41.825799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:41.825083 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:15:41.905058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:41.897154 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:41.905058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:41.897854 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:15:42.054117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:41.957535 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:42.054117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:41.958491 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:15:42.054117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:41.961487 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:15:42.054117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:41.997639 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:15:42.054117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:41.998147 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:42.054117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:41.998447 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:15:42.054117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.013108 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:42.054117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.013525 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:15:42.080056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.078117 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:15:42.080056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.078543 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:15:42.142058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.110251 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:42.142873 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.137202 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:42.175458 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.148109 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:15:42.281165 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.270134 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:42.295069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.282334 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:42.339771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.339572 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:15:42.364716 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.364651 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f6fad1-n7pln" Oct 30 21:15:42.466092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.456952 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1708326-0038-4cdb-8f8a-5e2577634a66-kube-api-access-rrhtk\" (UniqueName: \"kubernetes.io/projected/c1708326-0038-4cdb-8f8a-5e2577634a66-kube-api-access-rrhtk\") pod \"c1708326-0038-4cdb-8f8a-5e2577634a66\" (UID: \"c1708326-0038-4cdb-8f8a-5e2577634a66\") " Oct 30 21:15:42.476372 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.476338 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1708326-0038-4cdb-8f8a-5e2577634a66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1708326-0038-4cdb-8f8a-5e2577634a66-maas-api-xks-ca\") pod \"c1708326-0038-4cdb-8f8a-5e2577634a66\" (UID: \"c1708326-0038-4cdb-8f8a-5e2577634a66\") " Oct 30 21:15:42.476960 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.476936 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1708326-0038-4cdb-8f8a-5e2577634a66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1708326-0038-4cdb-8f8a-5e2577634a66-maas-api-openshift-ca\") pod \"c1708326-0038-4cdb-8f8a-5e2577634a66\" (UID: \"c1708326-0038-4cdb-8f8a-5e2577634a66\") " Oct 30 21:15:42.491055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.472515 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:15:42.491371 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.472560 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 21:15:42.495143 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.481211 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1708326-0038-4cdb-8f8a-5e2577634a66-maas-api-openshift-ca" pod "c1708326-0038-4cdb-8f8a-5e2577634a66" (UID: "c1708326-0038-4cdb-8f8a-5e2577634a66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:42.524120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.523986 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:42.524343 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.524022 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 21:15:42.559909 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.559859 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:15:42.606180 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.588154 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:42.606180 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.601158 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1708326-0038-4cdb-8f8a-5e2577634a66-maas-api-xks-ca" pod "c1708326-0038-4cdb-8f8a-5e2577634a66" (UID: "c1708326-0038-4cdb-8f8a-5e2577634a66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:15:42.606564 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.606539 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1708326-0038-4cdb-8f8a-5e2577634a66-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:42.606766 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.606725 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:42.692314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.692273 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1708326-0038-4cdb-8f8a-5e2577634a66-kube-api-access-rrhtk" pod "c1708326-0038-4cdb-8f8a-5e2577634a66" (UID: "c1708326-0038-4cdb-8f8a-5e2577634a66"). InnerVolumeSpecName "kube-api-access-rrhtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:42.708528 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.708493 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1708326-0038-4cdb-8f8a-5e2577634a66-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:42.708735 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.708719 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrhtk\" (UniqueName: \"kubernetes.io/projected/c1708326-0038-4cdb-8f8a-5e2577634a66-kube-api-access-rrhtk\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:42.721195 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.721151 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:42.730383 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.730352 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:15:42.856261 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.856154 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:42.885223 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:42.885180 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:15:43.099487 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:43.099441 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:15:43.169110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:43.168170 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:43.169397 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:43.169362 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 21:15:43.315250 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:43.315205 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:43.467398 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:43.467304 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 21:15:43.556054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:43.554119 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f6fad1-n7pln" Oct 30 21:15:43.614097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:43.571076 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:15:43.913021 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:43.912929 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:15:43.913434 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:43.913411 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:15:43.913695 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:43.913668 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89f6fad1-n7pln" event={"ID":"c1708326-0038-4cdb-8f8a-5e2577634a66","Type":"ContainerDied","Data":"549ee8138f5ae7c5c4013ea05329b0c68b914060acfd5d41fd55aef9c2452aaf"} Oct 30 21:15:43.998069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:43.994328 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="549ee8138f5ae7c5c4013ea05329b0c68b914060acfd5d41fd55aef9c2452aaf" Oct 30 21:15:43.998069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:43.642886 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:43.998069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:43.583863 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:43.998069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:43.862234 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:43.998069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:43.949682 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:43.998069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:43.994168 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:43.998069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:43.994266 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:15:44.089506 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:44.085184 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:44.486056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:44.485633 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:15:44.544425 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:44.493198 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:15:44.573057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:44.535073 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:44.599104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:44.536763 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:44.599104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:44.536793 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 21:15:44.599104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:44.536814 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:15:44.599104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:44.572352 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 21:15:44.599104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:44.572831 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:44.599104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:44.572857 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:15:44.599104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:44.572877 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 21:15:44.599104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:44.577594 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:15:44.715066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:44.715011 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:15:44.715722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:44.715683 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" podUID="8a80b31b-f4f7-47bc-abe7-27b0505919f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:45.251170 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:45.251122 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:15:45.274104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:45.261104 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:45.298200 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:45.268131 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:45.371082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:45.330511 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:45.371651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:45.330543 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:45.372120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:45.349785 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:45.391212 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:45.349824 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 21:15:45.509216 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:45.502475 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:45.580373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:45.553894 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:15:45.580857 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:45.580833 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:15:45.639954 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:45.639905 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:45.664109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:45.642763 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:45.682390 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:45.642792 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 21:15:45.686445 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:45.642814 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:45.690496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:45.690473 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:15:45.718798 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:45.642850 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 21:15:45.719349 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:45.642867 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:15:45.719825 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:45.719793 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:15:45.733096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:45.642833 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:15:46.256056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.244497 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7"] Oct 30 21:15:46.417524 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.417489 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c1708326-0038-4cdb-8f8a-5e2577634a66" containerName="revoke-keys" Oct 30 21:15:46.417743 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.417726 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1708326-0038-4cdb-8f8a-5e2577634a66" containerName="revoke-keys" Oct 30 21:15:46.418047 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.418015 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1708326-0038-4cdb-8f8a-5e2577634a66" containerName="revoke-keys" Oct 30 21:15:46.420189 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.420148 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 21:15:46.436069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.420341 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:46.451079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.256607 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:46.451079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.282234 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:46.451079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.372200 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:46.451079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.405289 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:46.451079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.445851 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 21:15:46.459069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.458517 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78"] Oct 30 21:15:46.504386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.464608 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:46.504386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.493244 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.603429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.598878 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:46.615062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.612395 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-token\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.615062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.612480 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87424b1f-d6bc-42f6-ad13-b85cc826043f-istiod-ca-cert\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.615062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.612513 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-credential-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:46.615062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.612557 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-podinfo\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.615062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.612593 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-data\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.615062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.612691 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-token\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:46.615062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.612733 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-envoy\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:46.615062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.612784 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-podinfo\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:46.615062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.612843 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83749c42-ef6a-4755-acbf-2166a604a6e6-istiod-ca-cert\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:46.615062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.612875 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-workload-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.615062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.612914 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-workload-certs\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.615062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.612978 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5c2w\" (UniqueName: \"kubernetes.io/projected/87424b1f-d6bc-42f6-ad13-b85cc826043f-kube-api-access-d5c2w\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.615062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.613012 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-workload-certs\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:46.615062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.613110 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-data\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:46.615062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.613146 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-credential-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.615062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.613201 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgn7q\" (UniqueName: \"kubernetes.io/projected/83749c42-ef6a-4755-acbf-2166a604a6e6-kube-api-access-rgn7q\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:46.615062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.613241 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-envoy\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.615062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.613311 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-workload-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:46.665435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.617138 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:46.665435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.647127 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8k468" podUID="97d86e50-7f90-4650-bef4-2a4cd91c1d61" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.37:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:46.665435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.654992 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7"] Oct 30 21:15:46.665435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.665097 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:46.665435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.665410 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:46.705723 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.705684 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.705652 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.714449 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-token\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.794771 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-envoy\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.794824 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-podinfo\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.794885 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83749c42-ef6a-4755-acbf-2166a604a6e6-istiod-ca-cert\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.794916 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-workload-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.794965 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-workload-certs\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.795049 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5c2w\" (UniqueName: \"kubernetes.io/projected/87424b1f-d6bc-42f6-ad13-b85cc826043f-kube-api-access-d5c2w\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.795086 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-workload-certs\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.795167 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-data\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.795199 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-credential-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.795258 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgn7q\" (UniqueName: \"kubernetes.io/projected/83749c42-ef6a-4755-acbf-2166a604a6e6-kube-api-access-rgn7q\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.795296 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-envoy\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.795363 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-workload-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.795416 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-token\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.795487 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87424b1f-d6bc-42f6-ad13-b85cc826043f-istiod-ca-cert\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.795520 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-credential-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.795569 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-podinfo\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.795599 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-data\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.796178 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-data\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.719181 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.797406 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-credential-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.720489 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.798736 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-workload-certs\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.748216 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:46.802057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.760086 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78"] Oct 30 21:15:46.883096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.876803 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-podinfo\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:46.883096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.877253 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-token\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:46.895116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.887358 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-envoy\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:46.937304 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.932602 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-workload-certs\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:47.000174 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:46.999468 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87424b1f-d6bc-42f6-ad13-b85cc826043f-istiod-ca-cert\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:47.037685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.003148 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-workload-socket\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:47.037685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.029839 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:15:47.047107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.046316 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5c2w\" (UniqueName: \"kubernetes.io/projected/87424b1f-d6bc-42f6-ad13-b85cc826043f-kube-api-access-d5c2w\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:47.116609 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.116564 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-data\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:47.124749 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.124715 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-envoy\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:47.125422 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.125397 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83749c42-ef6a-4755-acbf-2166a604a6e6-istiod-ca-cert\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:47.125825 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.125804 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-workload-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:47.145346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.145266 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-token\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:47.154073 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.153631 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-credential-socket\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:47.191207 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.172706 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-podinfo\") pod \"e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:47.344154 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.327778 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgn7q\" (UniqueName: \"kubernetes.io/projected/83749c42-ef6a-4755-acbf-2166a604a6e6-kube-api-access-rgn7q\") pod \"e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:47.433380 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.433347 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:47.433717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.433695 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:47.441386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.441338 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:47.441704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.441671 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:47.441902 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.441883 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:47.442101 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.442081 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:47.458720 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.458688 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:47.477144 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.466505 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 21:15:47.513225 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.513188 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8k468" podUID="97d86e50-7f90-4650-bef4-2a4cd91c1d61" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.37:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:15:47.663166 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.663135 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:15:47.666990 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.666959 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:15:47.667201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.667188 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:15:47.667320 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.667309 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:15:47.667425 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.667415 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:15:47.667524 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.667515 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:15:47.667631 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.667622 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:15:47.671051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.667722 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:15:47.671051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.667758 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:15:47.671051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.667787 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:15:47.676479 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.676460 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:15:47.682317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.682299 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:15:47.683972 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:47.683918 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:15:48.032848 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:15:48.032810 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87424b1f_d6bc_42f6_ad13_b85cc826043f.slice/crio-e075dea838309cf87a7e24fd8c55eb938043f7103ab444ad4594d21671e74e9c WatchSource:0}: Error finding container e075dea838309cf87a7e24fd8c55eb938043f7103ab444ad4594d21671e74e9c: Status 404 returned error can't find the container with id e075dea838309cf87a7e24fd8c55eb938043f7103ab444ad4594d21671e74e9c Oct 30 21:15:48.034632 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:48.034600 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78"] Oct 30 21:15:48.036550 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:48.036507 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:48.036663 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:48.036633 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:48.036775 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:48.036686 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:48.042332 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:48.042276 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7"] Oct 30 21:15:48.057657 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:48.057582 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:48.057760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:48.057662 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:48.057760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:48.057704 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:15:48.535751 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:48.535717 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" event={"ID":"87424b1f-d6bc-42f6-ad13-b85cc826043f","Type":"ContainerStarted","Data":"6fab60554de72477679321feab0725fd0923c88c86b71079abd8285f67784890"} Oct 30 21:15:48.535933 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:48.535769 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" event={"ID":"87424b1f-d6bc-42f6-ad13-b85cc826043f","Type":"ContainerStarted","Data":"e075dea838309cf87a7e24fd8c55eb938043f7103ab444ad4594d21671e74e9c"} Oct 30 21:15:48.538508 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:48.538481 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" event={"ID":"83749c42-ef6a-4755-acbf-2166a604a6e6","Type":"ContainerStarted","Data":"aeb066d5d6f84bcd64676d129486de4ca6d8acc4ce81f3031664e5414d943cde"} Oct 30 21:15:48.538657 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:48.538521 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" event={"ID":"83749c42-ef6a-4755-acbf-2166a604a6e6","Type":"ContainerStarted","Data":"15573d8f2bc0712408b9af2583d9025801c3d2b23b762d33b40fe5872b73e0b8"} Oct 30 21:15:48.546945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:48.546919 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:48.547111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:48.546961 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:48.590612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:48.590554 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" podStartSLOduration=3.590539373 podStartE2EDuration="3.590539373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:48.585888526 +0000 UTC m=+935.706962272" watchObservedRunningTime="2026-10-30 21:15:48.590539373 +0000 UTC m=+935.711613133" Oct 30 21:15:48.659818 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:48.659582 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" podStartSLOduration=3.659562932 podStartE2EDuration="3.659562932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:48.650453555 +0000 UTC m=+935.771527304" watchObservedRunningTime="2026-10-30 21:15:48.659562932 +0000 UTC m=+935.780636681" Oct 30 21:15:49.241808 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:49.241758 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08e40599-d69vc"] Oct 30 21:15:49.248192 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:49.248169 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e40599-d69vc" Oct 30 21:15:49.284076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:49.266208 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08e40599-d69vc"] Oct 30 21:15:49.299066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:49.287725 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:15:49.299066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:49.288413 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:15:49.341053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:49.341003 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f43dcb80-8b09-473c-b1a5-987225e5fa34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08e40599-d69vc\" (UID: \"f43dcb80-8b09-473c-b1a5-987225e5fa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e40599-d69vc" Oct 30 21:15:49.341250 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:49.341132 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f43dcb80-8b09-473c-b1a5-987225e5fa34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08e40599-d69vc\" (UID: \"f43dcb80-8b09-473c-b1a5-987225e5fa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e40599-d69vc" Oct 30 21:15:49.341250 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:49.341154 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z85d\" (UniqueName: \"kubernetes.io/projected/f43dcb80-8b09-473c-b1a5-987225e5fa34-kube-api-access-8z85d\") pod \"maas-api-revoke-sub-08e40599-d69vc\" (UID: \"f43dcb80-8b09-473c-b1a5-987225e5fa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e40599-d69vc" Oct 30 21:15:49.445056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:49.443092 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f43dcb80-8b09-473c-b1a5-987225e5fa34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08e40599-d69vc\" (UID: \"f43dcb80-8b09-473c-b1a5-987225e5fa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e40599-d69vc" Oct 30 21:15:49.445056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:49.443142 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z85d\" (UniqueName: \"kubernetes.io/projected/f43dcb80-8b09-473c-b1a5-987225e5fa34-kube-api-access-8z85d\") pod \"maas-api-revoke-sub-08e40599-d69vc\" (UID: \"f43dcb80-8b09-473c-b1a5-987225e5fa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e40599-d69vc" Oct 30 21:15:49.445056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:49.443426 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f43dcb80-8b09-473c-b1a5-987225e5fa34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08e40599-d69vc\" (UID: \"f43dcb80-8b09-473c-b1a5-987225e5fa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e40599-d69vc" Oct 30 21:15:49.445056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:49.444881 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f43dcb80-8b09-473c-b1a5-987225e5fa34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-08e40599-d69vc\" (UID: \"f43dcb80-8b09-473c-b1a5-987225e5fa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e40599-d69vc" Oct 30 21:15:49.451463 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:49.451431 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f43dcb80-8b09-473c-b1a5-987225e5fa34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-08e40599-d69vc\" (UID: \"f43dcb80-8b09-473c-b1a5-987225e5fa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e40599-d69vc" Oct 30 21:15:49.545105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:49.539790 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z85d\" (UniqueName: \"kubernetes.io/projected/f43dcb80-8b09-473c-b1a5-987225e5fa34-kube-api-access-8z85d\") pod \"maas-api-revoke-sub-08e40599-d69vc\" (UID: \"f43dcb80-8b09-473c-b1a5-987225e5fa34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e40599-d69vc" Oct 30 21:15:49.550345 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:49.549492 4926 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:15:49.550345 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:49.549551 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" podUID="83749c42-ef6a-4755-acbf-2166a604a6e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 21:15:49.550345 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:49.550103 4926 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:15:49.550345 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:49.550171 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" podUID="87424b1f-d6bc-42f6-ad13-b85cc826043f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 21:15:49.646511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:49.646475 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e40599-d69vc" Oct 30 21:15:50.130215 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:50.129302 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08e40599-d69vc"] Oct 30 21:15:50.258666 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:50.256313 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e38050-p9jsh"] Oct 30 21:15:50.265200 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:50.265171 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e38050-p9jsh" Oct 30 21:15:50.340202 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:50.340161 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e38050-p9jsh"] Oct 30 21:15:50.365305 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:50.365272 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn8mf\" (UniqueName: \"kubernetes.io/projected/d3f8b2fc-80e2-43d2-9509-8927c824e1d7-kube-api-access-hn8mf\") pod \"maas-api-revoke-sub-17e38050-p9jsh\" (UID: \"d3f8b2fc-80e2-43d2-9509-8927c824e1d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e38050-p9jsh" Oct 30 21:15:50.365497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:50.365316 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f8b2fc-80e2-43d2-9509-8927c824e1d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17e38050-p9jsh\" (UID: \"d3f8b2fc-80e2-43d2-9509-8927c824e1d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e38050-p9jsh" Oct 30 21:15:50.365497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:50.365409 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f8b2fc-80e2-43d2-9509-8927c824e1d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17e38050-p9jsh\" (UID: \"d3f8b2fc-80e2-43d2-9509-8927c824e1d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e38050-p9jsh" Oct 30 21:15:50.499057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:50.493240 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn8mf\" (UniqueName: \"kubernetes.io/projected/d3f8b2fc-80e2-43d2-9509-8927c824e1d7-kube-api-access-hn8mf\") pod \"maas-api-revoke-sub-17e38050-p9jsh\" (UID: \"d3f8b2fc-80e2-43d2-9509-8927c824e1d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e38050-p9jsh" Oct 30 21:15:50.499057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:50.493301 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f8b2fc-80e2-43d2-9509-8927c824e1d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17e38050-p9jsh\" (UID: \"d3f8b2fc-80e2-43d2-9509-8927c824e1d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e38050-p9jsh" Oct 30 21:15:50.499057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:50.493455 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f8b2fc-80e2-43d2-9509-8927c824e1d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17e38050-p9jsh\" (UID: \"d3f8b2fc-80e2-43d2-9509-8927c824e1d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e38050-p9jsh" Oct 30 21:15:50.518054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:50.517575 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f8b2fc-80e2-43d2-9509-8927c824e1d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17e38050-p9jsh\" (UID: \"d3f8b2fc-80e2-43d2-9509-8927c824e1d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e38050-p9jsh" Oct 30 21:15:50.541512 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:50.541452 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f8b2fc-80e2-43d2-9509-8927c824e1d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17e38050-p9jsh\" (UID: \"d3f8b2fc-80e2-43d2-9509-8927c824e1d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e38050-p9jsh" Oct 30 21:15:50.547047 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:50.543929 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn8mf\" (UniqueName: \"kubernetes.io/projected/d3f8b2fc-80e2-43d2-9509-8927c824e1d7-kube-api-access-hn8mf\") pod \"maas-api-revoke-sub-17e38050-p9jsh\" (UID: \"d3f8b2fc-80e2-43d2-9509-8927c824e1d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e38050-p9jsh" Oct 30 21:15:50.548051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:50.547671 4926 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:15:50.548051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:50.547710 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" podUID="83749c42-ef6a-4755-acbf-2166a604a6e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 21:15:50.550771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:50.550410 4926 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:15:50.550771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:50.550490 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" podUID="87424b1f-d6bc-42f6-ad13-b85cc826043f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 21:15:50.565914 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:50.565886 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e40599-d69vc" event={"ID":"f43dcb80-8b09-473c-b1a5-987225e5fa34","Type":"ContainerStarted","Data":"2a5be03890b7ee21e740f2299b428f7acd6f13b5dae627d95b74f0bd3288d421"} Oct 30 21:15:50.623600 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:50.623557 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e38050-p9jsh" Oct 30 21:15:51.049941 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:51.049914 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e38050-p9jsh"] Oct 30 21:15:51.547319 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:51.547284 4926 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:15:51.547707 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:51.547347 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" podUID="83749c42-ef6a-4755-acbf-2166a604a6e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 21:15:51.547707 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:51.547284 4926 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:15:51.547707 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:51.547417 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" podUID="87424b1f-d6bc-42f6-ad13-b85cc826043f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 21:15:51.638373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:51.638340 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e38050-p9jsh" event={"ID":"d3f8b2fc-80e2-43d2-9509-8927c824e1d7","Type":"ContainerStarted","Data":"02d44f3953159aeade60e7cb95c66001597aceb44b30d6cc503a71a0ac715251"} Oct 30 21:15:51.640878 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:51.640854 4926 generic.go:356] "Generic (PLEG): container finished" podID="f43dcb80-8b09-473c-b1a5-987225e5fa34" containerID="c68dec1b952c277b71954f641042d0adda85f5f26305b1d0ad866e4db66b6121" exitCode=0 Oct 30 21:15:51.640977 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:51.640919 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e40599-d69vc" event={"ID":"f43dcb80-8b09-473c-b1a5-987225e5fa34","Type":"ContainerDied","Data":"c68dec1b952c277b71954f641042d0adda85f5f26305b1d0ad866e4db66b6121"} Oct 30 21:15:52.547288 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:52.547234 4926 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:15:52.547288 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:52.547274 4926 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:15:52.547835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:52.547302 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" podUID="83749c42-ef6a-4755-acbf-2166a604a6e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 21:15:52.547835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:52.547314 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" podUID="87424b1f-d6bc-42f6-ad13-b85cc826043f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 21:15:52.653125 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:52.653085 4926 generic.go:356] "Generic (PLEG): container finished" podID="d3f8b2fc-80e2-43d2-9509-8927c824e1d7" containerID="bd144cb66f81eb4af57d8602a29f17da7dd86d4dcba768d70ac6aa68b47dd429" exitCode=0 Oct 30 21:15:52.653582 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:52.653560 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e38050-p9jsh" event={"ID":"d3f8b2fc-80e2-43d2-9509-8927c824e1d7","Type":"ContainerDied","Data":"bd144cb66f81eb4af57d8602a29f17da7dd86d4dcba768d70ac6aa68b47dd429"} Oct 30 21:15:53.130156 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:53.130078 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e40599-d69vc" Oct 30 21:15:53.249680 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:53.249645 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f43dcb80-8b09-473c-b1a5-987225e5fa34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f43dcb80-8b09-473c-b1a5-987225e5fa34-maas-api-xks-ca\") pod \"f43dcb80-8b09-473c-b1a5-987225e5fa34\" (UID: \"f43dcb80-8b09-473c-b1a5-987225e5fa34\") " Oct 30 21:15:53.249856 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:53.249712 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f43dcb80-8b09-473c-b1a5-987225e5fa34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f43dcb80-8b09-473c-b1a5-987225e5fa34-maas-api-openshift-ca\") pod \"f43dcb80-8b09-473c-b1a5-987225e5fa34\" (UID: \"f43dcb80-8b09-473c-b1a5-987225e5fa34\") " Oct 30 21:15:53.249856 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:53.249829 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f43dcb80-8b09-473c-b1a5-987225e5fa34-kube-api-access-8z85d\" (UniqueName: \"kubernetes.io/projected/f43dcb80-8b09-473c-b1a5-987225e5fa34-kube-api-access-8z85d\") pod \"f43dcb80-8b09-473c-b1a5-987225e5fa34\" (UID: \"f43dcb80-8b09-473c-b1a5-987225e5fa34\") " Oct 30 21:15:53.250163 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:53.250134 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f43dcb80-8b09-473c-b1a5-987225e5fa34-maas-api-openshift-ca" pod "f43dcb80-8b09-473c-b1a5-987225e5fa34" (UID: "f43dcb80-8b09-473c-b1a5-987225e5fa34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:53.250421 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:53.250400 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f43dcb80-8b09-473c-b1a5-987225e5fa34-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:53.251986 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:53.251958 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f43dcb80-8b09-473c-b1a5-987225e5fa34-maas-api-xks-ca" pod "f43dcb80-8b09-473c-b1a5-987225e5fa34" (UID: "f43dcb80-8b09-473c-b1a5-987225e5fa34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:15:53.252601 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:53.252279 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f43dcb80-8b09-473c-b1a5-987225e5fa34-kube-api-access-8z85d" pod "f43dcb80-8b09-473c-b1a5-987225e5fa34" (UID: "f43dcb80-8b09-473c-b1a5-987225e5fa34"). InnerVolumeSpecName "kube-api-access-8z85d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:53.352085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:53.351483 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f43dcb80-8b09-473c-b1a5-987225e5fa34-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:53.352085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:53.351512 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z85d\" (UniqueName: \"kubernetes.io/projected/f43dcb80-8b09-473c-b1a5-987225e5fa34-kube-api-access-8z85d\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:53.547470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:53.547378 4926 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:15:53.547470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:53.547438 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" podUID="87424b1f-d6bc-42f6-ad13-b85cc826043f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 21:15:53.547985 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:53.547378 4926 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:15:53.547985 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:53.547814 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" podUID="83749c42-ef6a-4755-acbf-2166a604a6e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 21:15:53.664872 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:53.664839 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e40599-d69vc" Oct 30 21:15:53.665061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:53.664880 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-08e40599-d69vc" event={"ID":"f43dcb80-8b09-473c-b1a5-987225e5fa34","Type":"ContainerDied","Data":"2a5be03890b7ee21e740f2299b428f7acd6f13b5dae627d95b74f0bd3288d421"} Oct 30 21:15:53.665061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:53.664935 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a5be03890b7ee21e740f2299b428f7acd6f13b5dae627d95b74f0bd3288d421" Oct 30 21:15:54.147206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:54.147129 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e38050-p9jsh" Oct 30 21:15:54.163286 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:54.163256 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3f8b2fc-80e2-43d2-9509-8927c824e1d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f8b2fc-80e2-43d2-9509-8927c824e1d7-maas-api-xks-ca\") pod \"d3f8b2fc-80e2-43d2-9509-8927c824e1d7\" (UID: \"d3f8b2fc-80e2-43d2-9509-8927c824e1d7\") " Oct 30 21:15:54.163424 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:54.163321 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3f8b2fc-80e2-43d2-9509-8927c824e1d7-kube-api-access-hn8mf\" (UniqueName: \"kubernetes.io/projected/d3f8b2fc-80e2-43d2-9509-8927c824e1d7-kube-api-access-hn8mf\") pod \"d3f8b2fc-80e2-43d2-9509-8927c824e1d7\" (UID: \"d3f8b2fc-80e2-43d2-9509-8927c824e1d7\") " Oct 30 21:15:54.163593 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:54.163573 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3f8b2fc-80e2-43d2-9509-8927c824e1d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f8b2fc-80e2-43d2-9509-8927c824e1d7-maas-api-openshift-ca\") pod \"d3f8b2fc-80e2-43d2-9509-8927c824e1d7\" (UID: \"d3f8b2fc-80e2-43d2-9509-8927c824e1d7\") " Oct 30 21:15:54.164008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:54.163982 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3f8b2fc-80e2-43d2-9509-8927c824e1d7-maas-api-openshift-ca" pod "d3f8b2fc-80e2-43d2-9509-8927c824e1d7" (UID: "d3f8b2fc-80e2-43d2-9509-8927c824e1d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:54.164302 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:54.164287 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3f8b2fc-80e2-43d2-9509-8927c824e1d7-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:54.165509 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:54.165486 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3f8b2fc-80e2-43d2-9509-8927c824e1d7-maas-api-xks-ca" pod "d3f8b2fc-80e2-43d2-9509-8927c824e1d7" (UID: "d3f8b2fc-80e2-43d2-9509-8927c824e1d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:15:54.165567 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:54.165526 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3f8b2fc-80e2-43d2-9509-8927c824e1d7-kube-api-access-hn8mf" pod "d3f8b2fc-80e2-43d2-9509-8927c824e1d7" (UID: "d3f8b2fc-80e2-43d2-9509-8927c824e1d7"). InnerVolumeSpecName "kube-api-access-hn8mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:54.270055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:54.266280 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3f8b2fc-80e2-43d2-9509-8927c824e1d7-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:54.270055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:54.266319 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn8mf\" (UniqueName: \"kubernetes.io/projected/d3f8b2fc-80e2-43d2-9509-8927c824e1d7-kube-api-access-hn8mf\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:54.547395 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:54.547358 4926 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Oct 30 21:15:54.547596 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:54.547416 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" podUID="83749c42-ef6a-4755-acbf-2166a604a6e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Oct 30 21:15:54.547596 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:54.547358 4926 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Oct 30 21:15:54.547596 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:54.547525 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" podUID="87424b1f-d6bc-42f6-ad13-b85cc826043f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Oct 30 21:15:54.677244 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:54.677215 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e38050-p9jsh" Oct 30 21:15:54.677394 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:54.677241 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17e38050-p9jsh" event={"ID":"d3f8b2fc-80e2-43d2-9509-8927c824e1d7","Type":"ContainerDied","Data":"02d44f3953159aeade60e7cb95c66001597aceb44b30d6cc503a71a0ac715251"} Oct 30 21:15:54.677394 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:54.677325 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02d44f3953159aeade60e7cb95c66001597aceb44b30d6cc503a71a0ac715251" Oct 30 21:15:55.062633 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.062600 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw"] Oct 30 21:15:55.063444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.063424 4926 cpu_manager.go:414] "containerMap: removing container" podUID="f43dcb80-8b09-473c-b1a5-987225e5fa34" containerName="revoke-keys" Oct 30 21:15:55.063497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.063449 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f43dcb80-8b09-473c-b1a5-987225e5fa34" containerName="revoke-keys" Oct 30 21:15:55.063497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.063487 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d3f8b2fc-80e2-43d2-9509-8927c824e1d7" containerName="revoke-keys" Oct 30 21:15:55.063565 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.063496 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3f8b2fc-80e2-43d2-9509-8927c824e1d7" containerName="revoke-keys" Oct 30 21:15:55.063692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.063671 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3f8b2fc-80e2-43d2-9509-8927c824e1d7" containerName="revoke-keys" Oct 30 21:15:55.063692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.063693 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="f43dcb80-8b09-473c-b1a5-987225e5fa34" containerName="revoke-keys" Oct 30 21:15:55.067473 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.067449 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw" Oct 30 21:15:55.083648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.081603 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f31a01e-712f-4e08-b7ef-f1fe343d368e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20ee3d8a-ch8mw\" (UID: \"7f31a01e-712f-4e08-b7ef-f1fe343d368e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw" Oct 30 21:15:55.083648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.082172 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f31a01e-712f-4e08-b7ef-f1fe343d368e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20ee3d8a-ch8mw\" (UID: \"7f31a01e-712f-4e08-b7ef-f1fe343d368e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw" Oct 30 21:15:55.083648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.082283 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4xnh\" (UniqueName: \"kubernetes.io/projected/7f31a01e-712f-4e08-b7ef-f1fe343d368e-kube-api-access-f4xnh\") pod \"maas-api-revoke-sub-20ee3d8a-ch8mw\" (UID: \"7f31a01e-712f-4e08-b7ef-f1fe343d368e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw" Oct 30 21:15:55.094014 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.093984 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:15:55.094182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.094095 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:15:55.098213 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.098185 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw"] Oct 30 21:15:55.189863 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.183888 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4xnh\" (UniqueName: \"kubernetes.io/projected/7f31a01e-712f-4e08-b7ef-f1fe343d368e-kube-api-access-f4xnh\") pod \"maas-api-revoke-sub-20ee3d8a-ch8mw\" (UID: \"7f31a01e-712f-4e08-b7ef-f1fe343d368e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw" Oct 30 21:15:55.189863 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.184107 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f31a01e-712f-4e08-b7ef-f1fe343d368e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20ee3d8a-ch8mw\" (UID: \"7f31a01e-712f-4e08-b7ef-f1fe343d368e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw" Oct 30 21:15:55.189863 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.184306 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f31a01e-712f-4e08-b7ef-f1fe343d368e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20ee3d8a-ch8mw\" (UID: \"7f31a01e-712f-4e08-b7ef-f1fe343d368e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw" Oct 30 21:15:55.189863 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.185606 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f31a01e-712f-4e08-b7ef-f1fe343d368e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20ee3d8a-ch8mw\" (UID: \"7f31a01e-712f-4e08-b7ef-f1fe343d368e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw" Oct 30 21:15:55.194365 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.194338 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f31a01e-712f-4e08-b7ef-f1fe343d368e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20ee3d8a-ch8mw\" (UID: \"7f31a01e-712f-4e08-b7ef-f1fe343d368e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw" Oct 30 21:15:55.205150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.205121 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4xnh\" (UniqueName: \"kubernetes.io/projected/7f31a01e-712f-4e08-b7ef-f1fe343d368e-kube-api-access-f4xnh\") pod \"maas-api-revoke-sub-20ee3d8a-ch8mw\" (UID: \"7f31a01e-712f-4e08-b7ef-f1fe343d368e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw" Oct 30 21:15:55.421972 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.420632 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw" Oct 30 21:15:55.852163 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.850491 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:55.852743 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.852722 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:55.853699 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.853681 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:15:55.878584 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:15:55.878545 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f31a01e_712f_4e08_b7ef_f1fe343d368e.slice/crio-30748b0b920b3af1ad5dcc0fd03cb6bfe2b23584c30a36922ba29c9ef7c515a0 WatchSource:0}: Error finding container 30748b0b920b3af1ad5dcc0fd03cb6bfe2b23584c30a36922ba29c9ef7c515a0: Status 404 returned error can't find the container with id 30748b0b920b3af1ad5dcc0fd03cb6bfe2b23584c30a36922ba29c9ef7c515a0 Oct 30 21:15:55.882361 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.882339 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw"] Oct 30 21:15:55.921085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.921027 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:55.921393 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.921372 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:55.922903 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:55.922877 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:15:56.735741 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:56.735698 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw" event={"ID":"7f31a01e-712f-4e08-b7ef-f1fe343d368e","Type":"ContainerStarted","Data":"70808953ead605130868f860042b3b777f8562e1f13adcd0c56b3800e73bc923"} Oct 30 21:15:56.735928 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:56.735761 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw" event={"ID":"7f31a01e-712f-4e08-b7ef-f1fe343d368e","Type":"ContainerStarted","Data":"30748b0b920b3af1ad5dcc0fd03cb6bfe2b23584c30a36922ba29c9ef7c515a0"} Oct 30 21:15:56.790275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:56.790211 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw" podStartSLOduration=1.790175515 podStartE2EDuration="1.790175515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:15:56.780226867 +0000 UTC m=+943.901300614" watchObservedRunningTime="2026-10-30 21:15:56.790175515 +0000 UTC m=+943.911249265" Oct 30 21:15:57.747144 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:57.747110 4926 generic.go:356] "Generic (PLEG): container finished" podID="7f31a01e-712f-4e08-b7ef-f1fe343d368e" containerID="70808953ead605130868f860042b3b777f8562e1f13adcd0c56b3800e73bc923" exitCode=0 Oct 30 21:15:57.747694 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:57.747191 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw" event={"ID":"7f31a01e-712f-4e08-b7ef-f1fe343d368e","Type":"ContainerDied","Data":"70808953ead605130868f860042b3b777f8562e1f13adcd0c56b3800e73bc923"} Oct 30 21:15:58.924269 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:58.924244 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw" Oct 30 21:15:58.949556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:58.949517 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f31a01e-712f-4e08-b7ef-f1fe343d368e-kube-api-access-f4xnh\" (UniqueName: \"kubernetes.io/projected/7f31a01e-712f-4e08-b7ef-f1fe343d368e-kube-api-access-f4xnh\") pod \"7f31a01e-712f-4e08-b7ef-f1fe343d368e\" (UID: \"7f31a01e-712f-4e08-b7ef-f1fe343d368e\") " Oct 30 21:15:58.949686 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:58.949607 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f31a01e-712f-4e08-b7ef-f1fe343d368e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f31a01e-712f-4e08-b7ef-f1fe343d368e-maas-api-openshift-ca\") pod \"7f31a01e-712f-4e08-b7ef-f1fe343d368e\" (UID: \"7f31a01e-712f-4e08-b7ef-f1fe343d368e\") " Oct 30 21:15:58.949783 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:58.949766 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f31a01e-712f-4e08-b7ef-f1fe343d368e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f31a01e-712f-4e08-b7ef-f1fe343d368e-maas-api-xks-ca\") pod \"7f31a01e-712f-4e08-b7ef-f1fe343d368e\" (UID: \"7f31a01e-712f-4e08-b7ef-f1fe343d368e\") " Oct 30 21:15:58.949943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:58.949917 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f31a01e-712f-4e08-b7ef-f1fe343d368e-maas-api-openshift-ca" pod "7f31a01e-712f-4e08-b7ef-f1fe343d368e" (UID: "7f31a01e-712f-4e08-b7ef-f1fe343d368e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:15:58.950397 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:58.950379 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f31a01e-712f-4e08-b7ef-f1fe343d368e-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:58.951616 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:58.951588 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f31a01e-712f-4e08-b7ef-f1fe343d368e-kube-api-access-f4xnh" pod "7f31a01e-712f-4e08-b7ef-f1fe343d368e" (UID: "7f31a01e-712f-4e08-b7ef-f1fe343d368e"). InnerVolumeSpecName "kube-api-access-f4xnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:15:58.951707 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:58.951684 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f31a01e-712f-4e08-b7ef-f1fe343d368e-maas-api-xks-ca" pod "7f31a01e-712f-4e08-b7ef-f1fe343d368e" (UID: "7f31a01e-712f-4e08-b7ef-f1fe343d368e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:15:59.051553 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:59.051511 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4xnh\" (UniqueName: \"kubernetes.io/projected/7f31a01e-712f-4e08-b7ef-f1fe343d368e-kube-api-access-f4xnh\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:59.051553 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:59.051544 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f31a01e-712f-4e08-b7ef-f1fe343d368e-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:15:59.764312 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:59.764285 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw" Oct 30 21:15:59.764460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:59.764315 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw" event={"ID":"7f31a01e-712f-4e08-b7ef-f1fe343d368e","Type":"ContainerDied","Data":"30748b0b920b3af1ad5dcc0fd03cb6bfe2b23584c30a36922ba29c9ef7c515a0"} Oct 30 21:15:59.764460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:15:59.764369 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30748b0b920b3af1ad5dcc0fd03cb6bfe2b23584c30a36922ba29c9ef7c515a0" Oct 30 21:16:04.462059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:04.459019 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:04.486937 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:04.486879 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:04.495419 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:04.495381 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:04.509278 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:04.509215 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:04.580907 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:04.580845 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:04.664055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:04.663582 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:04.737122 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:04.733950 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 21:16:04.827287 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:04.827239 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:05.066753 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:05.066648 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:05.264192 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:05.264144 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:05.381304 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:05.381202 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:05.477105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:05.477055 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 21:16:05.482179 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:05.482149 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:16:05.494743 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:05.494700 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:05.555179 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:05.555133 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:05.555826 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:05.555804 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:16:05.570198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:05.570160 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:05.572231 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:05.572208 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:16:05.615341 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:05.615280 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:05.619494 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:05.619466 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:16:05.716068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:05.709690 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:16:05.861789 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:05.861749 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:05.863220 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:05.863189 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:05.864589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:05.863684 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:16:06.113595 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:06.113485 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:16:06.119174 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:06.119146 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:16:06.180238 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:06.180169 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:06.301058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:06.274181 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:06.301058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:06.274839 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:16:06.332225 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:06.329553 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:16:06.521003 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:06.495254 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:06.521003 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:06.496664 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:06.521003 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:06.499495 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:16:06.565629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:06.565157 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:06.589068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:06.583939 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:06.618180 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:06.618121 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 21:16:06.672177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:06.672126 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:06.721200 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:06.721157 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 21:16:06.747058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:06.723098 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:06.838107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:06.812730 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:16:06.870360 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:06.868110 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:07.210061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:07.204123 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:07.291885 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:07.291824 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:07.321276 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:07.321216 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:16:07.399128 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:07.399072 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 21:16:07.508117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:07.507506 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:16:07.634571 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:07.634529 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:07.684198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:07.684152 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:07.685620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:07.685581 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:07.727205 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:07.727158 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 21:16:07.728199 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:07.728169 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:07.773631 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:07.753599 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:07.895293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:07.894346 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 21:16:07.930465 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:07.930434 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:16:07.930761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:07.930744 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:16:07.930925 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:07.930912 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:16:07.931082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:07.931068 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:16:07.931222 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:07.931211 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:16:07.931342 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:07.931331 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:16:08.036108 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:16:08.034885 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:16:08.307120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:08.279898 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 21:16:08.324202 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:08.324156 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:08.325215 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:08.325192 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:16:08.437053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:08.400588 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:08.486057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:08.445736 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:16:08.513056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:08.512207 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:16:08.642693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:08.641852 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:08.686297 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:08.686254 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:08.686658 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:08.686527 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 21:16:08.687607 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:08.687586 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:16:08.722277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:08.722251 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:16:08.726332 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:08.726299 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:16:08.727715 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:08.727688 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:08.728675 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:08.728653 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:08.797993 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:08.797844 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:08.917360 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:08.917306 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 21:16:09.131380 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:09.131344 4926 patch_prober.go:29] interesting pod/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:16:09.131687 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:09.131637 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" podUID="5397fa31-83d0-4500-94bb-16e1848efe2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:09.300857 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:09.280359 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:09.300857 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:09.297123 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:09.373256 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:09.373211 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:16:09.406781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:09.388918 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:16:09.445930 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:09.445886 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 21:16:09.470149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:09.469957 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:16:09.533156 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:09.532171 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:16:09.533439 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:09.533413 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:16:09.534119 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:09.534084 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:09.535150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:09.534370 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:16:09.607277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:09.606260 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:16:09.692934 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:09.692904 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:16:09.746055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:09.729127 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:09.746055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:09.728021 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:09.839137 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:09.839104 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:16:09.845184 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:09.845157 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:16:15.301671 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:15.301628 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq"] Oct 30 21:16:15.309700 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:15.309659 4926 cpu_manager.go:414] "containerMap: removing container" podUID="7f31a01e-712f-4e08-b7ef-f1fe343d368e" containerName="revoke-keys" Oct 30 21:16:15.309700 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:15.309694 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f31a01e-712f-4e08-b7ef-f1fe343d368e" containerName="revoke-keys" Oct 30 21:16:15.309972 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:15.309918 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f31a01e-712f-4e08-b7ef-f1fe343d368e" containerName="revoke-keys" Oct 30 21:16:15.336706 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:15.336654 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq"] Oct 30 21:16:15.336915 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:15.336789 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq" Oct 30 21:16:15.350665 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:15.350621 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:16:15.350665 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:15.350635 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:16:15.498140 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:15.498096 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kghv\" (UniqueName: \"kubernetes.io/projected/6cbd47fc-81f6-46b9-8916-76b6e5cf7002-kube-api-access-2kghv\") pod \"maas-api-revoke-sub-c14822df-7m7pq\" (UID: \"6cbd47fc-81f6-46b9-8916-76b6e5cf7002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq" Oct 30 21:16:15.498436 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:15.498409 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cbd47fc-81f6-46b9-8916-76b6e5cf7002-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c14822df-7m7pq\" (UID: \"6cbd47fc-81f6-46b9-8916-76b6e5cf7002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq" Oct 30 21:16:15.498507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:15.498467 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cbd47fc-81f6-46b9-8916-76b6e5cf7002-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c14822df-7m7pq\" (UID: \"6cbd47fc-81f6-46b9-8916-76b6e5cf7002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq" Oct 30 21:16:15.599999 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:15.599906 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cbd47fc-81f6-46b9-8916-76b6e5cf7002-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c14822df-7m7pq\" (UID: \"6cbd47fc-81f6-46b9-8916-76b6e5cf7002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq" Oct 30 21:16:15.599999 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:15.599955 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cbd47fc-81f6-46b9-8916-76b6e5cf7002-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c14822df-7m7pq\" (UID: \"6cbd47fc-81f6-46b9-8916-76b6e5cf7002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq" Oct 30 21:16:15.600297 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:15.600073 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kghv\" (UniqueName: \"kubernetes.io/projected/6cbd47fc-81f6-46b9-8916-76b6e5cf7002-kube-api-access-2kghv\") pod \"maas-api-revoke-sub-c14822df-7m7pq\" (UID: \"6cbd47fc-81f6-46b9-8916-76b6e5cf7002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq" Oct 30 21:16:15.600663 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:15.600636 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cbd47fc-81f6-46b9-8916-76b6e5cf7002-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c14822df-7m7pq\" (UID: \"6cbd47fc-81f6-46b9-8916-76b6e5cf7002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq" Oct 30 21:16:15.602366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:15.602341 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cbd47fc-81f6-46b9-8916-76b6e5cf7002-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c14822df-7m7pq\" (UID: \"6cbd47fc-81f6-46b9-8916-76b6e5cf7002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq" Oct 30 21:16:15.620468 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:15.620436 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kghv\" (UniqueName: \"kubernetes.io/projected/6cbd47fc-81f6-46b9-8916-76b6e5cf7002-kube-api-access-2kghv\") pod \"maas-api-revoke-sub-c14822df-7m7pq\" (UID: \"6cbd47fc-81f6-46b9-8916-76b6e5cf7002\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq" Oct 30 21:16:15.653358 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:15.653323 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq" Oct 30 21:16:15.901669 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:15.901571 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp"] Oct 30 21:16:15.906901 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:15.906874 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:15.939023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:15.938525 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp"] Oct 30 21:16:15.971658 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:16:15.971627 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cbd47fc_81f6_46b9_8916_76b6e5cf7002.slice/crio-d04c06275bd27987a604bea5285102dfcd346363aa67ca2b4ca41841a47d72b0 WatchSource:0}: Error finding container d04c06275bd27987a604bea5285102dfcd346363aa67ca2b4ca41841a47d72b0: Status 404 returned error can't find the container with id d04c06275bd27987a604bea5285102dfcd346363aa67ca2b4ca41841a47d72b0 Oct 30 21:16:15.984933 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:15.984907 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq"] Oct 30 21:16:16.006931 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.006902 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f026f40-c790-4639-8975-98e56da80ce9-istio-token\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.007102 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.006960 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6924b\" (UniqueName: \"kubernetes.io/projected/3f026f40-c790-4639-8975-98e56da80ce9-kube-api-access-6924b\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.007182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.007114 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-credential-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.007182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.007143 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f026f40-c790-4639-8975-98e56da80ce9-istio-podinfo\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.007319 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.007263 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-workload-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.007319 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.007305 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f026f40-c790-4639-8975-98e56da80ce9-istiod-ca-cert\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.007444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.007353 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-istio-envoy\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.007444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.007438 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-workload-certs\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.007554 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.007458 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-istio-data\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.108446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.108408 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-credential-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.108631 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.108460 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f026f40-c790-4639-8975-98e56da80ce9-istio-podinfo\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.108631 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.108526 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-workload-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.108631 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.108561 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f026f40-c790-4639-8975-98e56da80ce9-istiod-ca-cert\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.108631 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.108594 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-istio-envoy\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.108834 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.108669 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-workload-certs\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.108834 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.108744 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-istio-data\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.108943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.108847 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f026f40-c790-4639-8975-98e56da80ce9-istio-token\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.108943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.108905 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6924b\" (UniqueName: \"kubernetes.io/projected/3f026f40-c790-4639-8975-98e56da80ce9-kube-api-access-6924b\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.109062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.108972 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-credential-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.109062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.108995 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-workload-socket\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.109556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.109327 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-istio-data\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.109556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.109434 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-workload-certs\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.109740 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.109708 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f026f40-c790-4639-8975-98e56da80ce9-istiod-ca-cert\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.167731 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.167698 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-istio-envoy\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.168255 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.168222 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f026f40-c790-4639-8975-98e56da80ce9-istio-token\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.168417 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.168315 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6924b\" (UniqueName: \"kubernetes.io/projected/3f026f40-c790-4639-8975-98e56da80ce9-kube-api-access-6924b\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.168574 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.168522 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f026f40-c790-4639-8975-98e56da80ce9-istio-podinfo\") pod \"e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.229948 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.229914 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:16.559835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.559780 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp"] Oct 30 21:16:16.560568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.560531 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:16:16.560648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.560605 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:16:16.560648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.560636 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:16:16.808322 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.808217 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" event={"ID":"3f026f40-c790-4639-8975-98e56da80ce9","Type":"ContainerStarted","Data":"4c721600dc428d4b2935c3d0fb16dce12e3a367fe25151615922d49e6296720d"} Oct 30 21:16:16.808322 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.808293 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" event={"ID":"3f026f40-c790-4639-8975-98e56da80ce9","Type":"ContainerStarted","Data":"09a2cb783d2e9636b6f869ce8f09a69aca2b2dbcd39e43f2f4ea5df42fbf2698"} Oct 30 21:16:16.810527 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.810502 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq" event={"ID":"6cbd47fc-81f6-46b9-8916-76b6e5cf7002","Type":"ContainerStarted","Data":"3edef04b1358351b220e330f724d89ad620b4c28f6819e53eb7c67415dbb45ec"} Oct 30 21:16:16.810675 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.810539 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq" event={"ID":"6cbd47fc-81f6-46b9-8916-76b6e5cf7002","Type":"ContainerStarted","Data":"d04c06275bd27987a604bea5285102dfcd346363aa67ca2b4ca41841a47d72b0"} Oct 30 21:16:16.885932 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.885869 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" podStartSLOduration=1.8858496790000001 podStartE2EDuration="1.885849679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:16.865529095 +0000 UTC m=+963.986602842" watchObservedRunningTime="2026-10-30 21:16:16.885849679 +0000 UTC m=+964.006923425" Oct 30 21:16:16.915109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:16.912458 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq" podStartSLOduration=1.912442792 podStartE2EDuration="1.912442792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:16.90982807 +0000 UTC m=+964.030901816" watchObservedRunningTime="2026-10-30 21:16:16.912442792 +0000 UTC m=+964.033516537" Oct 30 21:16:17.230915 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:17.230870 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:17.232758 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:17.232723 4926 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:16:17.232913 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:17.232786 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" podUID="3f026f40-c790-4639-8975-98e56da80ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 21:16:17.822013 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:17.821974 4926 generic.go:356] "Generic (PLEG): container finished" podID="6cbd47fc-81f6-46b9-8916-76b6e5cf7002" containerID="3edef04b1358351b220e330f724d89ad620b4c28f6819e53eb7c67415dbb45ec" exitCode=0 Oct 30 21:16:17.822534 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:17.822065 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq" event={"ID":"6cbd47fc-81f6-46b9-8916-76b6e5cf7002","Type":"ContainerDied","Data":"3edef04b1358351b220e330f724d89ad620b4c28f6819e53eb7c67415dbb45ec"} Oct 30 21:16:18.230456 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:18.230420 4926 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:16:18.230606 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:18.230476 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" podUID="3f026f40-c790-4639-8975-98e56da80ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 21:16:19.064955 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:19.064927 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq" Oct 30 21:16:19.217014 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:19.216968 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cbd47fc-81f6-46b9-8916-76b6e5cf7002-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cbd47fc-81f6-46b9-8916-76b6e5cf7002-maas-api-xks-ca\") pod \"6cbd47fc-81f6-46b9-8916-76b6e5cf7002\" (UID: \"6cbd47fc-81f6-46b9-8916-76b6e5cf7002\") " Oct 30 21:16:19.217277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:19.217196 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cbd47fc-81f6-46b9-8916-76b6e5cf7002-kube-api-access-2kghv\" (UniqueName: \"kubernetes.io/projected/6cbd47fc-81f6-46b9-8916-76b6e5cf7002-kube-api-access-2kghv\") pod \"6cbd47fc-81f6-46b9-8916-76b6e5cf7002\" (UID: \"6cbd47fc-81f6-46b9-8916-76b6e5cf7002\") " Oct 30 21:16:19.217277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:19.217251 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cbd47fc-81f6-46b9-8916-76b6e5cf7002-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cbd47fc-81f6-46b9-8916-76b6e5cf7002-maas-api-openshift-ca\") pod \"6cbd47fc-81f6-46b9-8916-76b6e5cf7002\" (UID: \"6cbd47fc-81f6-46b9-8916-76b6e5cf7002\") " Oct 30 21:16:19.219279 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:19.219244 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cbd47fc-81f6-46b9-8916-76b6e5cf7002-maas-api-openshift-ca" pod "6cbd47fc-81f6-46b9-8916-76b6e5cf7002" (UID: "6cbd47fc-81f6-46b9-8916-76b6e5cf7002"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:16:19.220849 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:19.220823 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cbd47fc-81f6-46b9-8916-76b6e5cf7002-kube-api-access-2kghv" pod "6cbd47fc-81f6-46b9-8916-76b6e5cf7002" (UID: "6cbd47fc-81f6-46b9-8916-76b6e5cf7002"). InnerVolumeSpecName "kube-api-access-2kghv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:16:19.227397 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:19.227372 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cbd47fc-81f6-46b9-8916-76b6e5cf7002-maas-api-xks-ca" pod "6cbd47fc-81f6-46b9-8916-76b6e5cf7002" (UID: "6cbd47fc-81f6-46b9-8916-76b6e5cf7002"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:16:19.233061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:19.233009 4926 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:16:19.233147 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:19.233076 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" podUID="3f026f40-c790-4639-8975-98e56da80ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 21:16:19.319635 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:19.319594 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cbd47fc-81f6-46b9-8916-76b6e5cf7002-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:16:19.319635 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:19.319637 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kghv\" (UniqueName: \"kubernetes.io/projected/6cbd47fc-81f6-46b9-8916-76b6e5cf7002-kube-api-access-2kghv\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:16:19.319843 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:19.319654 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cbd47fc-81f6-46b9-8916-76b6e5cf7002-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:16:19.841245 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:19.841195 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq" event={"ID":"6cbd47fc-81f6-46b9-8916-76b6e5cf7002","Type":"ContainerDied","Data":"d04c06275bd27987a604bea5285102dfcd346363aa67ca2b4ca41841a47d72b0"} Oct 30 21:16:19.841431 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:19.841262 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d04c06275bd27987a604bea5285102dfcd346363aa67ca2b4ca41841a47d72b0" Oct 30 21:16:19.841431 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:19.841224 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq" Oct 30 21:16:20.231450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:20.231409 4926 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:16:20.231951 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:20.231483 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" podUID="3f026f40-c790-4639-8975-98e56da80ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 21:16:21.230690 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:21.230647 4926 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:16:21.230866 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:21.230727 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" podUID="3f026f40-c790-4639-8975-98e56da80ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 21:16:22.230778 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:22.230746 4926 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:16:22.231179 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:22.230800 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" podUID="3f026f40-c790-4639-8975-98e56da80ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 21:16:23.230426 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:23.230392 4926 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Oct 30 21:16:23.230595 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:23.230446 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" podUID="3f026f40-c790-4639-8975-98e56da80ce9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Oct 30 21:16:24.236105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:24.236070 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:24.236609 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:24.236586 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:24.237473 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:24.237449 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:16:30.822599 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.822561 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj"] Oct 30 21:16:30.823246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.823231 4926 cpu_manager.go:414] "containerMap: removing container" podUID="6cbd47fc-81f6-46b9-8916-76b6e5cf7002" containerName="revoke-keys" Oct 30 21:16:30.823246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.823247 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cbd47fc-81f6-46b9-8916-76b6e5cf7002" containerName="revoke-keys" Oct 30 21:16:30.823422 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.823410 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cbd47fc-81f6-46b9-8916-76b6e5cf7002" containerName="revoke-keys" Oct 30 21:16:30.827365 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.827344 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.839904 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.839872 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v"] Oct 30 21:16:30.843637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.843610 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqbjn\" (UniqueName: \"kubernetes.io/projected/37bf8b31-c305-4605-b262-f7f0aa6d2644-kube-api-access-tqbjn\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.843768 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.843650 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-workload-certs\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.843768 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.843672 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-workload-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.843885 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.843818 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37bf8b31-c305-4605-b262-f7f0aa6d2644-istiod-ca-cert\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.843949 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.843928 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-data\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.844077 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.844013 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-podinfo\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.844160 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.844141 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-envoy\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.844216 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.844203 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-credential-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.844273 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.844254 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-token\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.845523 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.845500 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:30.846980 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.846958 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj"] Oct 30 21:16:30.921715 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.921684 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v"] Oct 30 21:16:30.945461 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.945431 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-credential-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.945654 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.945490 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-token\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.945654 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.945527 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-podinfo\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:30.945654 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.945552 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-envoy\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:30.945654 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.945579 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-workload-certs\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:30.945947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.945760 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-credential-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:30.945947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.945830 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqbjn\" (UniqueName: \"kubernetes.io/projected/37bf8b31-c305-4605-b262-f7f0aa6d2644-kube-api-access-tqbjn\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.945947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.945870 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-workload-certs\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.945947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.945902 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-workload-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.945947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.945929 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istiod-ca-cert\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:30.945947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.945902 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-credential-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.946327 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.946026 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37bf8b31-c305-4605-b262-f7f0aa6d2644-istiod-ca-cert\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.946327 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.946164 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-data\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.946327 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.946210 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-podinfo\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.946327 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.946242 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-data\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:30.946327 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.946303 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-envoy\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.946601 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.946329 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcqtd\" (UniqueName: \"kubernetes.io/projected/6f7d1086-9e69-4241-ad92-cc49bf1bca22-kube-api-access-tcqtd\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:30.946601 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.946369 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-workload-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:30.946601 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.946402 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-token\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:30.946601 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.946578 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-workload-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.946817 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.946770 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-data\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.946876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.946827 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-workload-certs\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.946931 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.946867 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37bf8b31-c305-4605-b262-f7f0aa6d2644-istiod-ca-cert\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.949112 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.949082 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-envoy\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.949300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.949284 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-podinfo\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.970175 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.970149 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqbjn\" (UniqueName: \"kubernetes.io/projected/37bf8b31-c305-4605-b262-f7f0aa6d2644-kube-api-access-tqbjn\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.980098 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.980071 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-token\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:30.983104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.983075 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e642b143-pt4f5"] Oct 30 21:16:30.988523 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:30.988502 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e642b143-pt4f5" Oct 30 21:16:31.002536 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.002374 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:16:31.002536 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.002506 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:16:31.004138 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.004111 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e642b143-pt4f5"] Oct 30 21:16:31.047572 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.047536 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-data\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:31.047759 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.047596 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcqtd\" (UniqueName: \"kubernetes.io/projected/6f7d1086-9e69-4241-ad92-cc49bf1bca22-kube-api-access-tcqtd\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:31.047759 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.047625 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-workload-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:31.047759 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.047657 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-token\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:31.047759 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.047712 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjhzh\" (UniqueName: \"kubernetes.io/projected/f395eae6-6fea-4108-9022-e52ed87ca24d-kube-api-access-sjhzh\") pod \"maas-api-revoke-sub-e642b143-pt4f5\" (UID: \"f395eae6-6fea-4108-9022-e52ed87ca24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e642b143-pt4f5" Oct 30 21:16:31.048108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.047774 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-podinfo\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:31.048108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.047800 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-envoy\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:31.048108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.047827 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-workload-certs\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:31.048108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.047894 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-credential-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:31.048108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.047974 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istiod-ca-cert\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:31.048108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.048003 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f395eae6-6fea-4108-9022-e52ed87ca24d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e642b143-pt4f5\" (UID: \"f395eae6-6fea-4108-9022-e52ed87ca24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e642b143-pt4f5" Oct 30 21:16:31.048449 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.048201 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f395eae6-6fea-4108-9022-e52ed87ca24d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e642b143-pt4f5\" (UID: \"f395eae6-6fea-4108-9022-e52ed87ca24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e642b143-pt4f5" Oct 30 21:16:31.048449 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.048386 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-workload-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:31.048561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.048535 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-credential-socket\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:31.048665 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.048639 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-workload-certs\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:31.048732 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.048001 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-data\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:31.049228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.049202 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istiod-ca-cert\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:31.050755 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.050734 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-envoy\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:31.051273 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.051254 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-podinfo\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:31.077696 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.077618 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-token\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:31.083073 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.079938 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcqtd\" (UniqueName: \"kubernetes.io/projected/6f7d1086-9e69-4241-ad92-cc49bf1bca22-kube-api-access-tcqtd\") pod \"e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:31.144103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.144073 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:31.149819 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.149790 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f395eae6-6fea-4108-9022-e52ed87ca24d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e642b143-pt4f5\" (UID: \"f395eae6-6fea-4108-9022-e52ed87ca24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e642b143-pt4f5" Oct 30 21:16:31.149955 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.149900 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f395eae6-6fea-4108-9022-e52ed87ca24d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e642b143-pt4f5\" (UID: \"f395eae6-6fea-4108-9022-e52ed87ca24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e642b143-pt4f5" Oct 30 21:16:31.150057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.150020 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjhzh\" (UniqueName: \"kubernetes.io/projected/f395eae6-6fea-4108-9022-e52ed87ca24d-kube-api-access-sjhzh\") pod \"maas-api-revoke-sub-e642b143-pt4f5\" (UID: \"f395eae6-6fea-4108-9022-e52ed87ca24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e642b143-pt4f5" Oct 30 21:16:31.150583 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.150561 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f395eae6-6fea-4108-9022-e52ed87ca24d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e642b143-pt4f5\" (UID: \"f395eae6-6fea-4108-9022-e52ed87ca24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e642b143-pt4f5" Oct 30 21:16:31.152154 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.152136 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f395eae6-6fea-4108-9022-e52ed87ca24d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e642b143-pt4f5\" (UID: \"f395eae6-6fea-4108-9022-e52ed87ca24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e642b143-pt4f5" Oct 30 21:16:31.169350 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.169326 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjhzh\" (UniqueName: \"kubernetes.io/projected/f395eae6-6fea-4108-9022-e52ed87ca24d-kube-api-access-sjhzh\") pod \"maas-api-revoke-sub-e642b143-pt4f5\" (UID: \"f395eae6-6fea-4108-9022-e52ed87ca24d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e642b143-pt4f5" Oct 30 21:16:31.213343 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.213310 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:31.323766 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.318463 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e642b143-pt4f5" Oct 30 21:16:31.487450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.485340 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj"] Oct 30 21:16:31.491604 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:16:31.491280 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37bf8b31_c305_4605_b262_f7f0aa6d2644.slice/crio-9a5b86b38ba65061ff818afd65b982250def54a82398b7759ec40cb20efda610 WatchSource:0}: Error finding container 9a5b86b38ba65061ff818afd65b982250def54a82398b7759ec40cb20efda610: Status 404 returned error can't find the container with id 9a5b86b38ba65061ff818afd65b982250def54a82398b7759ec40cb20efda610 Oct 30 21:16:31.513991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.511637 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:16:31.513991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.511723 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:16:31.513991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.511767 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:16:31.564950 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.564923 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v"] Oct 30 21:16:31.571125 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.571087 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:16:31.571332 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.571310 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:16:31.571474 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.571457 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:16:31.705754 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.705722 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e642b143-pt4f5"] Oct 30 21:16:31.958559 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.958518 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" event={"ID":"6f7d1086-9e69-4241-ad92-cc49bf1bca22","Type":"ContainerStarted","Data":"049e80c22f213e9950d7554314014368899e69eebe4b63e39cbcf837546027c3"} Oct 30 21:16:31.959194 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.959165 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" event={"ID":"6f7d1086-9e69-4241-ad92-cc49bf1bca22","Type":"ContainerStarted","Data":"634ac37cfd9e2606ed28375db90f085a63d7e28079bd97ea2776d5041624d9fa"} Oct 30 21:16:31.961363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.961335 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e642b143-pt4f5" event={"ID":"f395eae6-6fea-4108-9022-e52ed87ca24d","Type":"ContainerStarted","Data":"ed7bca15d6628a3e3408cb7b8d8b82e06ba0cc298d7d39bb8c478ce7515d2ab3"} Oct 30 21:16:31.963410 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.963387 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" event={"ID":"37bf8b31-c305-4605-b262-f7f0aa6d2644","Type":"ContainerStarted","Data":"7fe18c40a71b364d1f49be81f1cc870d034725b5a29b299bde30419e5e8af284"} Oct 30 21:16:31.963510 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:31.963430 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" event={"ID":"37bf8b31-c305-4605-b262-f7f0aa6d2644","Type":"ContainerStarted","Data":"9a5b86b38ba65061ff818afd65b982250def54a82398b7759ec40cb20efda610"} Oct 30 21:16:32.016064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:32.015337 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" podStartSLOduration=2.015317398 podStartE2EDuration="2.015317398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:32.007532439 +0000 UTC m=+979.128606185" watchObservedRunningTime="2026-10-30 21:16:32.015317398 +0000 UTC m=+979.136391144" Oct 30 21:16:32.145499 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:32.145409 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:32.147244 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:32.147217 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:16:32.147455 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:32.147432 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" podUID="37bf8b31-c305-4605-b262-f7f0aa6d2644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 21:16:32.214444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:32.214347 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:32.215907 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:32.215574 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:16:32.215907 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:32.215621 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" podUID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:16:33.048900 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:33.048811 4926 generic.go:356] "Generic (PLEG): container finished" podID="f395eae6-6fea-4108-9022-e52ed87ca24d" containerID="18023aa7e318917871c42f51f704d063dc98f022c262fbb691044d8ea36cebc2" exitCode=0 Oct 30 21:16:33.053297 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:33.053271 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e642b143-pt4f5" event={"ID":"f395eae6-6fea-4108-9022-e52ed87ca24d","Type":"ContainerDied","Data":"18023aa7e318917871c42f51f704d063dc98f022c262fbb691044d8ea36cebc2"} Oct 30 21:16:33.118057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:33.109446 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" podStartSLOduration=3.109425015 podStartE2EDuration="3.109425015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:32.062282856 +0000 UTC m=+979.183356603" watchObservedRunningTime="2026-10-30 21:16:33.109425015 +0000 UTC m=+980.230498762" Oct 30 21:16:33.151075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:33.150939 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:16:33.151075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:33.150992 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" podUID="37bf8b31-c305-4605-b262-f7f0aa6d2644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 21:16:33.221551 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:33.221511 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:16:33.243208 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:33.243142 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" podUID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:16:34.144477 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.144441 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:16:34.144875 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.144492 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" podUID="37bf8b31-c305-4605-b262-f7f0aa6d2644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 21:16:34.216504 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.215634 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:16:34.216504 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.215688 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" podUID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:16:34.443014 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.442983 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e642b143-pt4f5" Oct 30 21:16:34.548823 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.548782 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f395eae6-6fea-4108-9022-e52ed87ca24d-kube-api-access-sjhzh\" (UniqueName: \"kubernetes.io/projected/f395eae6-6fea-4108-9022-e52ed87ca24d-kube-api-access-sjhzh\") pod \"f395eae6-6fea-4108-9022-e52ed87ca24d\" (UID: \"f395eae6-6fea-4108-9022-e52ed87ca24d\") " Oct 30 21:16:34.549078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.548939 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f395eae6-6fea-4108-9022-e52ed87ca24d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f395eae6-6fea-4108-9022-e52ed87ca24d-maas-api-xks-ca\") pod \"f395eae6-6fea-4108-9022-e52ed87ca24d\" (UID: \"f395eae6-6fea-4108-9022-e52ed87ca24d\") " Oct 30 21:16:34.549078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.549003 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f395eae6-6fea-4108-9022-e52ed87ca24d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f395eae6-6fea-4108-9022-e52ed87ca24d-maas-api-openshift-ca\") pod \"f395eae6-6fea-4108-9022-e52ed87ca24d\" (UID: \"f395eae6-6fea-4108-9022-e52ed87ca24d\") " Oct 30 21:16:34.550135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.550104 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f395eae6-6fea-4108-9022-e52ed87ca24d-maas-api-openshift-ca" pod "f395eae6-6fea-4108-9022-e52ed87ca24d" (UID: "f395eae6-6fea-4108-9022-e52ed87ca24d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:16:34.553232 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.553191 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f395eae6-6fea-4108-9022-e52ed87ca24d-kube-api-access-sjhzh" pod "f395eae6-6fea-4108-9022-e52ed87ca24d" (UID: "f395eae6-6fea-4108-9022-e52ed87ca24d"). InnerVolumeSpecName "kube-api-access-sjhzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:16:34.556924 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.556893 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f395eae6-6fea-4108-9022-e52ed87ca24d-maas-api-xks-ca" pod "f395eae6-6fea-4108-9022-e52ed87ca24d" (UID: "f395eae6-6fea-4108-9022-e52ed87ca24d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:16:34.651319 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.651231 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjhzh\" (UniqueName: \"kubernetes.io/projected/f395eae6-6fea-4108-9022-e52ed87ca24d-kube-api-access-sjhzh\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:16:34.651530 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.651510 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f395eae6-6fea-4108-9022-e52ed87ca24d-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:16:34.651634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.651621 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f395eae6-6fea-4108-9022-e52ed87ca24d-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:16:34.837988 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.837935 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp"] Oct 30 21:16:34.839905 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.839877 4926 cpu_manager.go:414] "containerMap: removing container" podUID="f395eae6-6fea-4108-9022-e52ed87ca24d" containerName="revoke-keys" Oct 30 21:16:34.841677 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.841658 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f395eae6-6fea-4108-9022-e52ed87ca24d" containerName="revoke-keys" Oct 30 21:16:34.842066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.842047 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="f395eae6-6fea-4108-9022-e52ed87ca24d" containerName="revoke-keys" Oct 30 21:16:34.851566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.849444 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" Oct 30 21:16:34.851566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.850702 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp"] Oct 30 21:16:34.863735 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.863628 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4b62455c-serving-cert\"" Oct 30 21:16:34.964865 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.964825 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4820b8d6-9171-4e76-a572-7e8d2007280c-maas-api-tls\") pod \"maas-api-e2e-ait-4b62455c-794d66845b-4wfvp\" (UID: \"4820b8d6-9171-4e76-a572-7e8d2007280c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" Oct 30 21:16:34.965058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.964942 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dvcf\" (UniqueName: \"kubernetes.io/projected/4820b8d6-9171-4e76-a572-7e8d2007280c-kube-api-access-6dvcf\") pod \"maas-api-e2e-ait-4b62455c-794d66845b-4wfvp\" (UID: \"4820b8d6-9171-4e76-a572-7e8d2007280c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" Oct 30 21:16:34.965058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.965013 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4820b8d6-9171-4e76-a572-7e8d2007280c-openshift-service-ca\") pod \"maas-api-e2e-ait-4b62455c-794d66845b-4wfvp\" (UID: \"4820b8d6-9171-4e76-a572-7e8d2007280c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" Oct 30 21:16:34.965207 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:34.965141 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4820b8d6-9171-4e76-a572-7e8d2007280c-metrics-certs\") pod \"maas-api-e2e-ait-4b62455c-794d66845b-4wfvp\" (UID: \"4820b8d6-9171-4e76-a572-7e8d2007280c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" Oct 30 21:16:35.094058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:35.066614 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4820b8d6-9171-4e76-a572-7e8d2007280c-metrics-certs\") pod \"maas-api-e2e-ait-4b62455c-794d66845b-4wfvp\" (UID: \"4820b8d6-9171-4e76-a572-7e8d2007280c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" Oct 30 21:16:35.112067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:35.094659 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4820b8d6-9171-4e76-a572-7e8d2007280c-maas-api-tls\") pod \"maas-api-e2e-ait-4b62455c-794d66845b-4wfvp\" (UID: \"4820b8d6-9171-4e76-a572-7e8d2007280c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" Oct 30 21:16:35.159715 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:35.159682 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dvcf\" (UniqueName: \"kubernetes.io/projected/4820b8d6-9171-4e76-a572-7e8d2007280c-kube-api-access-6dvcf\") pod \"maas-api-e2e-ait-4b62455c-794d66845b-4wfvp\" (UID: \"4820b8d6-9171-4e76-a572-7e8d2007280c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" Oct 30 21:16:35.160371 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:35.160329 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4820b8d6-9171-4e76-a572-7e8d2007280c-openshift-service-ca\") pod \"maas-api-e2e-ait-4b62455c-794d66845b-4wfvp\" (UID: \"4820b8d6-9171-4e76-a572-7e8d2007280c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" Oct 30 21:16:35.161236 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:16:35.113348 4926 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-serving-cert: secret "maas-api-e2e-ait-4b62455c-serving-cert" not found Oct 30 21:16:35.161342 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:16:35.161286 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4820b8d6-9171-4e76-a572-7e8d2007280c-maas-api-tls podName:4820b8d6-9171-4e76-a572-7e8d2007280c nodeName:}" failed. No retries permitted until 2026-10-30 21:16:35.661267877 +0000 UTC m=+982.782341601 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4820b8d6-9171-4e76-a572-7e8d2007280c-maas-api-tls") pod "maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" (UID: "4820b8d6-9171-4e76-a572-7e8d2007280c") : secret "maas-api-e2e-ait-4b62455c-serving-cert" not found Oct 30 21:16:35.161689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:35.145098 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:16:35.161689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:35.161588 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" podUID="37bf8b31-c305-4605-b262-f7f0aa6d2644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 21:16:35.161689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:35.144666 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4820b8d6-9171-4e76-a572-7e8d2007280c-metrics-certs\") pod \"maas-api-e2e-ait-4b62455c-794d66845b-4wfvp\" (UID: \"4820b8d6-9171-4e76-a572-7e8d2007280c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" Oct 30 21:16:35.162175 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:35.162151 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4820b8d6-9171-4e76-a572-7e8d2007280c-openshift-service-ca\") pod \"maas-api-e2e-ait-4b62455c-794d66845b-4wfvp\" (UID: \"4820b8d6-9171-4e76-a572-7e8d2007280c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" Oct 30 21:16:35.195149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:35.195118 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dvcf\" (UniqueName: \"kubernetes.io/projected/4820b8d6-9171-4e76-a572-7e8d2007280c-kube-api-access-6dvcf\") pod \"maas-api-e2e-ait-4b62455c-794d66845b-4wfvp\" (UID: \"4820b8d6-9171-4e76-a572-7e8d2007280c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" Oct 30 21:16:35.237152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:35.209759 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e642b143-pt4f5" event={"ID":"f395eae6-6fea-4108-9022-e52ed87ca24d","Type":"ContainerDied","Data":"ed7bca15d6628a3e3408cb7b8d8b82e06ba0cc298d7d39bb8c478ce7515d2ab3"} Oct 30 21:16:35.237152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:35.217216 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed7bca15d6628a3e3408cb7b8d8b82e06ba0cc298d7d39bb8c478ce7515d2ab3" Oct 30 21:16:35.237152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:35.215462 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:16:35.237152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:35.217272 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" podUID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:16:35.237152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:35.209834 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e642b143-pt4f5" Oct 30 21:16:35.681056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:35.679264 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4820b8d6-9171-4e76-a572-7e8d2007280c-maas-api-tls\") pod \"maas-api-e2e-ait-4b62455c-794d66845b-4wfvp\" (UID: \"4820b8d6-9171-4e76-a572-7e8d2007280c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" Oct 30 21:16:35.701056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:35.700113 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4820b8d6-9171-4e76-a572-7e8d2007280c-maas-api-tls\") pod \"maas-api-e2e-ait-4b62455c-794d66845b-4wfvp\" (UID: \"4820b8d6-9171-4e76-a572-7e8d2007280c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" Oct 30 21:16:35.845059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:35.844742 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" Oct 30 21:16:36.148802 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:36.148699 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:16:36.149104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:36.149068 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" podUID="37bf8b31-c305-4605-b262-f7f0aa6d2644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 21:16:36.233225 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:36.232701 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:16:36.233225 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:36.232766 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" podUID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:16:36.840610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:36.840492 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp"] Oct 30 21:16:37.145064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:37.144958 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:16:37.145064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:37.145044 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" podUID="37bf8b31-c305-4605-b262-f7f0aa6d2644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 21:16:37.220055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:37.219408 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:16:37.220055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:37.219472 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" podUID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:16:37.350427 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:37.350383 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" event={"ID":"4820b8d6-9171-4e76-a572-7e8d2007280c","Type":"ContainerStarted","Data":"7ac1586fa94fc3df7a6819b02f9d451a8cd35ef7902a619dabcad8cd10b59e04"} Oct 30 21:16:37.540593 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:37.540547 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:16:37.618101 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:16:37.608209 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:16:38.145160 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.145126 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:16:38.145398 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.145186 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" podUID="37bf8b31-c305-4605-b262-f7f0aa6d2644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 21:16:38.217758 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.217720 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:16:38.218008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.217979 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" podUID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:16:38.356062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.353826 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm"] Oct 30 21:16:38.359836 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.359810 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.361629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.361597 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm"] Oct 30 21:16:38.375670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.375637 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" event={"ID":"4820b8d6-9171-4e76-a572-7e8d2007280c","Type":"ContainerStarted","Data":"53e2db74f0d08fe62f8dab8f7bfc5ead2cf2addbc179051ac8e279eab434542c"} Oct 30 21:16:38.376236 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.376214 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" Oct 30 21:16:38.376476 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.376456 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1f954ca9-openshift-default-dockercfg-m944h\"" Oct 30 21:16:38.547068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.545943 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-credential-socket\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.547068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.546004 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-workload-socket\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.547068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.546064 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-podinfo\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.547068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.546143 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-envoy\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.547068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.546165 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-data\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.547068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.546236 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istiod-ca-cert\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.547068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.546295 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-token\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.547068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.546333 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82f7j\" (UniqueName: \"kubernetes.io/projected/3f4943da-3bdc-4e2c-900a-5c286a52bd09-kube-api-access-82f7j\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.547068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.546402 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-workload-certs\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.583054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.582819 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" podStartSLOduration=3.83000629 podStartE2EDuration="4.582798854s" totalImagesPullingTime="752.792564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:38.577806625 +0000 UTC m=+985.698880377" watchObservedRunningTime="2026-10-30 21:16:38.582798854 +0000 UTC m=+985.703872601" Oct 30 21:16:38.652074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.651981 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-workload-socket\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.652074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.652069 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-podinfo\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.652330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.652122 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-envoy\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.652330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.652146 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-data\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.652330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.652274 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istiod-ca-cert\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.652483 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.652333 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-token\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.652483 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.652369 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82f7j\" (UniqueName: \"kubernetes.io/projected/3f4943da-3bdc-4e2c-900a-5c286a52bd09-kube-api-access-82f7j\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.652483 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.652436 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-workload-certs\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.652674 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.652656 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-credential-socket\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.653195 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.653166 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-credential-socket\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.653854 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.653832 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-workload-socket\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.670755 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.670717 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istiod-ca-cert\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.672130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.671867 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-podinfo\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.673018 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.672821 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-workload-certs\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.673443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.673418 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-data\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.680159 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.680133 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-envoy\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.698076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.696658 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-token\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.709061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.708143 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82f7j\" (UniqueName: \"kubernetes.io/projected/3f4943da-3bdc-4e2c-900a-5c286a52bd09-kube-api-access-82f7j\") pod \"e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:38.984351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:38.984303 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:39.152212 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:39.150539 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 21:16:39.152212 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:39.150611 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" podUID="37bf8b31-c305-4605-b262-f7f0aa6d2644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 21:16:39.331110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:39.328486 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:16:39.331110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:39.328561 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" podUID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:16:39.936276 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:16:39.918641 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4943da_3bdc_4e2c_900a_5c286a52bd09.slice/crio-0472767544c1e187351a8794d763ccf4ff5bf60bfee3800c57cc656185399bac WatchSource:0}: Error finding container 0472767544c1e187351a8794d763ccf4ff5bf60bfee3800c57cc656185399bac: Status 404 returned error can't find the container with id 0472767544c1e187351a8794d763ccf4ff5bf60bfee3800c57cc656185399bac Oct 30 21:16:40.022614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:40.018571 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:16:40.022614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:40.018675 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:16:40.022614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:40.018720 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:16:40.052092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:40.051450 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm"] Oct 30 21:16:40.219926 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:40.216520 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 21:16:40.219926 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:40.216602 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" podUID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 21:16:40.339745 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:40.304118 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:40.513057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:40.512172 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 21:16:40.645070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:40.634116 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:40.645471 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:40.645428 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 21:16:40.723089 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:40.717615 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 21:16:40.734571 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:40.729828 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:40.736965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:40.736926 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 21:16:40.798168 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:40.798072 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" event={"ID":"3f4943da-3bdc-4e2c-900a-5c286a52bd09","Type":"ContainerStarted","Data":"0472767544c1e187351a8794d763ccf4ff5bf60bfee3800c57cc656185399bac"} Oct 30 21:16:40.848165 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:40.848118 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:41.196057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.195143 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:16:41.196057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.195215 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" podUID="37bf8b31-c305-4605-b262-f7f0aa6d2644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:41.459113 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.448323 4926 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:16:41.459113 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.448391 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" podUID="94ddad94-5958-43f7-b1fc-4c66a3117a29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:41.459113 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.449652 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:41.483055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.479346 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 21:16:41.515056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.513166 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:41.515056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.513501 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:16:41.515056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.514508 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:16:41.650055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.648143 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:41.650055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.648559 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:16:41.650311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.650171 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:41.650497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.650469 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:41.650771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.650750 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:16:41.651658 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.651632 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8k468" podUID="97d86e50-7f90-4650-bef4-2a4cd91c1d61" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.37:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:41.773110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.754280 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:41.773110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.754699 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:16:41.817157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.793105 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:41.823629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.823591 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:41.872568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.872511 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:41.873309 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.873285 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:16:41.907060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.904409 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 21:16:41.907060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:41.905147 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:42.143696 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:42.143595 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:42.345055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:42.265431 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:42.385058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:42.285121 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:16:42.730592 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:42.673459 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" podUID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:42.836059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:42.704340 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 21:16:42.884102 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:42.359349 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" event={"ID":"3f4943da-3bdc-4e2c-900a-5c286a52bd09","Type":"ContainerStarted","Data":"d515d4e57a4ce92641851cdda30fd632453315210af5f0c51d2860e068d38550"} Oct 30 21:16:42.884546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:42.704445 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded" Oct 30 21:16:42.884722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:42.704466 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 21:16:42.911568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:42.293178 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:16:42.912290 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:42.912240 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" podUID="37bf8b31-c305-4605-b262-f7f0aa6d2644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:43.000116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:42.999914 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 21:16:43.023117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:43.008234 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 21:16:43.023117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:43.014205 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:16:43.041103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:42.776662 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 21:16:43.041103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:42.776741 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": context deadline exceeded" Oct 30 21:16:43.041103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:43.024990 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:16:43.041103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:43.025199 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:16:43.041103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:42.704487 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:16:43.101796 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:42.704518 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:16:43.101796 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:42.704539 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 21:16:43.101796 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:43.051273 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:16:43.101796 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:43.051362 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:16:43.101796 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:43.051390 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:43.101796 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:43.085268 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 21:16:43.204118 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:43.177763 4926 patch_prober.go:29] interesting pod/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:16:43.204118 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:43.177827 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" podUID="3f4943da-3bdc-4e2c-900a-5c286a52bd09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 21:16:43.257162 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:42.704397 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 21:16:43.257162 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:43.233689 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:16:43.441620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:43.437712 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded" Oct 30 21:16:43.441620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:43.439322 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 21:16:43.441620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:43.440999 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 21:16:43.450863 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:43.442334 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:16:43.522369 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:43.522265 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded" Oct 30 21:16:43.854218 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:43.854117 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:43.933056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:43.885168 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:43.933402 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:43.933365 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:16:43.965167 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:43.965118 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" podUID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:43.965664 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:43.960316 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:16:43.965858 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:43.965836 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" podUID="37bf8b31-c305-4605-b262-f7f0aa6d2644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:44.004888 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.004843 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 21:16:44.019360 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.010235 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 21:16:44.022193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.020098 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:44.054157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.050627 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:44.067352 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.055425 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:16:44.067883 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.055459 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:16:44.098057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.073387 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:44.202058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.200530 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:44.236191 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.236143 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:44.380059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.362734 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:44.380425 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.380398 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:16:44.380590 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.380568 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:16:44.380704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.380692 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn"] Oct 30 21:16:44.384532 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.384500 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:16:44.501545 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.501445 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" podUID="8a80b31b-f4f7-47bc-abe7-27b0505919f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:44.502118 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.444498 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:16:44.502526 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.440821 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 21:16:44.509053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.502744 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn"] Oct 30 21:16:44.523251 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.523216 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:16:44.523502 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.523486 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:16:44.550067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.444528 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 21:16:44.550658 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.502943 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn" Oct 30 21:16:44.613209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.605771 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:16:44.613209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.605930 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:44.613209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.607181 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:16:44.613209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.607269 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:44.613209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.607475 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:44.613209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.608282 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dd8d\" (UniqueName: \"kubernetes.io/projected/8491486d-cb78-40a4-ab36-7ea5ece8a5a0-kube-api-access-2dd8d\") pod \"maas-api-revoke-sub-c49ffb9b-j9dpn\" (UID: \"8491486d-cb78-40a4-ab36-7ea5ece8a5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn" Oct 30 21:16:44.613209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.608556 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8491486d-cb78-40a4-ab36-7ea5ece8a5a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c49ffb9b-j9dpn\" (UID: \"8491486d-cb78-40a4-ab36-7ea5ece8a5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn" Oct 30 21:16:44.613209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.608617 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8491486d-cb78-40a4-ab36-7ea5ece8a5a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c49ffb9b-j9dpn\" (UID: \"8491486d-cb78-40a4-ab36-7ea5ece8a5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn" Oct 30 21:16:44.632521 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.628556 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:16:44.723210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.723176 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" Oct 30 21:16:44.735183 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.724666 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8491486d-cb78-40a4-ab36-7ea5ece8a5a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c49ffb9b-j9dpn\" (UID: \"8491486d-cb78-40a4-ab36-7ea5ece8a5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn" Oct 30 21:16:44.735492 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.735466 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8491486d-cb78-40a4-ab36-7ea5ece8a5a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c49ffb9b-j9dpn\" (UID: \"8491486d-cb78-40a4-ab36-7ea5ece8a5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn" Oct 30 21:16:44.735886 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.735862 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dd8d\" (UniqueName: \"kubernetes.io/projected/8491486d-cb78-40a4-ab36-7ea5ece8a5a0-kube-api-access-2dd8d\") pod \"maas-api-revoke-sub-c49ffb9b-j9dpn\" (UID: \"8491486d-cb78-40a4-ab36-7ea5ece8a5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn" Oct 30 21:16:44.757124 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.726130 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8491486d-cb78-40a4-ab36-7ea5ece8a5a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c49ffb9b-j9dpn\" (UID: \"8491486d-cb78-40a4-ab36-7ea5ece8a5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn" Oct 30 21:16:44.793064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.775871 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": context deadline exceeded" Oct 30 21:16:44.897059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.864384 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:44.908252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:44.908211 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:45.125172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.120221 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:45.125172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.120606 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:45.125172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.120903 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:45.129757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.129477 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8491486d-cb78-40a4-ab36-7ea5ece8a5a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c49ffb9b-j9dpn\" (UID: \"8491486d-cb78-40a4-ab36-7ea5ece8a5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn" Oct 30 21:16:45.231058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.155229 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:16:45.231058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.160096 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:16:45.231058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.160470 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 21:16:45.231058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.160755 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:16:45.231058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.188130 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dd8d\" (UniqueName: \"kubernetes.io/projected/8491486d-cb78-40a4-ab36-7ea5ece8a5a0-kube-api-access-2dd8d\") pod \"maas-api-revoke-sub-c49ffb9b-j9dpn\" (UID: \"8491486d-cb78-40a4-ab36-7ea5ece8a5a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn" Oct 30 21:16:45.232593 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.232543 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:16:45.232842 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.232809 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" podUID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:45.234025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.233991 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 21:16:45.262587 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.242152 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:45.342059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.294165 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:45.342513 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.342471 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:16:45.342734 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.315747 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn" Oct 30 21:16:45.343108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.308389 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:16:45.344460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.344432 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" podUID="a3061a9a-e0c1-46c9-914b-a5aba158ec6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:45.636546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.541504 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:45.636546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.542412 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 21:16:45.636546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.551085 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:45.777716 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.777646 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" podStartSLOduration=7.77762504 podStartE2EDuration="7.77762504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:43.274760486 +0000 UTC m=+990.395834234" watchObservedRunningTime="2026-10-30 21:16:45.77762504 +0000 UTC m=+992.898698787" Oct 30 21:16:45.896149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:45.870459 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:46.056391 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:46.056347 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 21:16:46.135058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:46.123345 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:46.172124 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:46.130557 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 21:16:46.209055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:46.130615 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 21:16:46.222057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:46.160493 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:46.256426 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:46.181214 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:46.376059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:46.212559 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 21:16:46.376059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:46.212603 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:16:46.376059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:46.250575 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:46.376059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:46.252489 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:46.376059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:46.252522 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:46.376059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:46.254793 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:16:46.376059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:46.358385 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" podUID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:46.557117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:46.545609 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:16:46.557117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:46.546003 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded" Oct 30 21:16:46.557117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:46.553657 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:46.557117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:46.553748 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 21:16:46.604472 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:46.603845 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 21:16:47.006613 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:47.006580 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:16:47.027384 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:47.027338 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 21:16:47.157289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:47.157235 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 21:16:47.276188 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:47.273263 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 21:16:47.292932 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:47.292878 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:47.327299 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:47.280146 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:47.332731 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:47.332701 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:16:47.356303 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:47.356259 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:47.356641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:47.356610 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:16:47.362173 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:47.358124 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:47.362173 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:47.358870 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:47.672116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:47.608769 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:16:47.672116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:47.608832 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" podUID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:47.672116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:47.634782 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:47.672116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:47.635176 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:16:47.672116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:47.659349 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 21:16:47.756736 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:47.753129 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:48.022203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.022109 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz"] Oct 30 21:16:48.192086 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.122497 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 21:16:48.192086 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.157955 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:48.222469 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.222436 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:48.270060 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:16:48.238135 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8491486d_cb78_40a4_ab36_7ea5ece8a5a0.slice/crio-605c0491eda5ad52409f86407a9e76ab28a6db66f14e20ed7c84d3ea8aae2c97 WatchSource:0}: Error finding container 605c0491eda5ad52409f86407a9e76ab28a6db66f14e20ed7c84d3ea8aae2c97: Status 404 returned error can't find the container with id 605c0491eda5ad52409f86407a9e76ab28a6db66f14e20ed7c84d3ea8aae2c97 Oct 30 21:16:48.282836 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.265773 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 21:16:48.282836 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.266522 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 21:16:48.282836 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.266561 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 21:16:48.282836 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.266642 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wmcm5\"" Oct 30 21:16:48.283311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.283276 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:48.300234 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.300194 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:48.338579 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.336513 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:48.444110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.442416 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 21:16:48.518107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.448746 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:48.556410 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.556309 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s4mz\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:48.556726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.556703 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s4mz\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:48.557062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.557021 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d775j\" (UniqueName: \"kubernetes.io/projected/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-kube-api-access-d775j\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s4mz\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:48.557247 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.557218 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s4mz\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:48.557404 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.557388 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s4mz\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:48.557607 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.557592 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s4mz\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:48.557790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.557777 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s4mz\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:48.606057 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:16:48.558251 4926 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60ed799_e6b9_464d_87d3_9f46bdfc7fcc.slice\": RecentStats: unable to find data in memory cache]" Oct 30 21:16:48.620208 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.600087 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:48.620577 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.600124 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:16:48.621972 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.600408 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz"] Oct 30 21:16:48.643305 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.643265 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:48.645906 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.645857 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:48.654222 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.646203 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 21:16:48.678564 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.661068 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s4mz\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:48.678564 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.661146 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s4mz\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:48.678564 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.661275 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d775j\" (UniqueName: \"kubernetes.io/projected/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-kube-api-access-d775j\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s4mz\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:48.678564 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.661335 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s4mz\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:48.678564 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.661376 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s4mz\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:48.678564 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.661457 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s4mz\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:48.678564 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.661509 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s4mz\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:48.680628 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.680574 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:16:48.680937 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.680822 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" podUID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:48.720723 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.714326 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s4mz\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:48.811126 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.760546 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s4mz\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:48.811126 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:48.761198 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 21:16:49.051085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.041569 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s4mz\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:49.051085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.042074 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s4mz\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:49.051085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.042326 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s4mz\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:49.051085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.042735 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s4mz\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:49.112518 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.112480 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d775j\" (UniqueName: \"kubernetes.io/projected/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-kube-api-access-d775j\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s4mz\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:49.258059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.250591 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:49.258059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.250971 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 21:16:49.281414 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.279120 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:49.309061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.284361 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:49.309889 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.309846 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:49.406113 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.357267 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:49.406113 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.361794 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:49.471094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.457157 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:49.472414 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.472377 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:16:49.715533 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.516446 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn" event={"ID":"8491486d-cb78-40a4-ab36-7ea5ece8a5a0","Type":"ContainerStarted","Data":"605c0491eda5ad52409f86407a9e76ab28a6db66f14e20ed7c84d3ea8aae2c97"} Oct 30 21:16:49.739940 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.616769 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:49.740446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.665672 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:49.783862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.665776 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:49.783862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.665807 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:16:49.783862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.665837 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:49.783862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.665865 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:49.783862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.732422 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:16:49.783862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.764501 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" podUID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:49.783862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.764877 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:49.836428 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:49.744971 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:16:50.241228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.237935 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:50.270513 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.269066 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:16:50.303266 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.303217 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 21:16:50.304321 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.304287 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 21:16:50.304833 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.304805 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:50.332770 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.332729 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:16:50.341059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.333913 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:16:50.368977 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.368943 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:16:50.403129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.402148 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:16:50.403129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.402242 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:16:50.409393 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.409356 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:16:50.419646 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.419609 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:16:50.452181 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.445209 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:16:50.520716 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.519362 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:50.575064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.568527 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:16:50.575064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.568632 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:16:50.579394 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.579361 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:16:50.591882 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.591431 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:16:50.591882 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.591499 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:16:50.595396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.594758 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:16:50.596840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.596821 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:16:50.598378 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.598362 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:16:50.630244 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.630214 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:16:50.635697 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.635645 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn" event={"ID":"8491486d-cb78-40a4-ab36-7ea5ece8a5a0","Type":"ContainerStarted","Data":"c4f80056f277ad1febbdf8f8a9022de41b043217353d1f8cc65f1ed3d7794d6d"} Oct 30 21:16:50.682180 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.682142 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn"] Oct 30 21:16:50.781896 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:50.781369 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz"] Oct 30 21:16:51.164161 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:51.164101 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn" podStartSLOduration=9.164081941 podStartE2EDuration="9.164081941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:51.160088527 +0000 UTC m=+998.281162272" watchObservedRunningTime="2026-10-30 21:16:51.164081941 +0000 UTC m=+998.285155688" Oct 30 21:16:51.720995 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:51.710923 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" event={"ID":"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc","Type":"ContainerStarted","Data":"0c61e970451a6344e31bf98be2d57f3e004e98e43cae539b0665b75e801cf853"} Oct 30 21:16:51.720995 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:51.713338 4926 generic.go:356] "Generic (PLEG): container finished" podID="8491486d-cb78-40a4-ab36-7ea5ece8a5a0" containerID="c4f80056f277ad1febbdf8f8a9022de41b043217353d1f8cc65f1ed3d7794d6d" exitCode=0 Oct 30 21:16:51.720995 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:51.714704 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn" event={"ID":"8491486d-cb78-40a4-ab36-7ea5ece8a5a0","Type":"ContainerDied","Data":"c4f80056f277ad1febbdf8f8a9022de41b043217353d1f8cc65f1ed3d7794d6d"} Oct 30 21:16:52.932179 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:52.932146 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn" Oct 30 21:16:53.087181 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:53.085559 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8491486d-cb78-40a4-ab36-7ea5ece8a5a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8491486d-cb78-40a4-ab36-7ea5ece8a5a0-maas-api-openshift-ca\") pod \"8491486d-cb78-40a4-ab36-7ea5ece8a5a0\" (UID: \"8491486d-cb78-40a4-ab36-7ea5ece8a5a0\") " Oct 30 21:16:53.087181 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:53.085688 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8491486d-cb78-40a4-ab36-7ea5ece8a5a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8491486d-cb78-40a4-ab36-7ea5ece8a5a0-maas-api-xks-ca\") pod \"8491486d-cb78-40a4-ab36-7ea5ece8a5a0\" (UID: \"8491486d-cb78-40a4-ab36-7ea5ece8a5a0\") " Oct 30 21:16:53.087181 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:53.085847 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8491486d-cb78-40a4-ab36-7ea5ece8a5a0-kube-api-access-2dd8d\" (UniqueName: \"kubernetes.io/projected/8491486d-cb78-40a4-ab36-7ea5ece8a5a0-kube-api-access-2dd8d\") pod \"8491486d-cb78-40a4-ab36-7ea5ece8a5a0\" (UID: \"8491486d-cb78-40a4-ab36-7ea5ece8a5a0\") " Oct 30 21:16:53.087181 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:53.085965 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8491486d-cb78-40a4-ab36-7ea5ece8a5a0-maas-api-openshift-ca" pod "8491486d-cb78-40a4-ab36-7ea5ece8a5a0" (UID: "8491486d-cb78-40a4-ab36-7ea5ece8a5a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:16:53.087181 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:53.086550 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8491486d-cb78-40a4-ab36-7ea5ece8a5a0-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:16:53.091263 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:53.091236 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8491486d-cb78-40a4-ab36-7ea5ece8a5a0-maas-api-xks-ca" pod "8491486d-cb78-40a4-ab36-7ea5ece8a5a0" (UID: "8491486d-cb78-40a4-ab36-7ea5ece8a5a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:16:53.095863 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:53.095831 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8491486d-cb78-40a4-ab36-7ea5ece8a5a0-kube-api-access-2dd8d" pod "8491486d-cb78-40a4-ab36-7ea5ece8a5a0" (UID: "8491486d-cb78-40a4-ab36-7ea5ece8a5a0"). InnerVolumeSpecName "kube-api-access-2dd8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:16:53.188651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:53.188573 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8491486d-cb78-40a4-ab36-7ea5ece8a5a0-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:16:53.188651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:53.188621 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dd8d\" (UniqueName: \"kubernetes.io/projected/8491486d-cb78-40a4-ab36-7ea5ece8a5a0-kube-api-access-2dd8d\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:16:53.736418 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:53.736385 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn" Oct 30 21:16:53.736659 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:53.736443 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn" event={"ID":"8491486d-cb78-40a4-ab36-7ea5ece8a5a0","Type":"ContainerDied","Data":"605c0491eda5ad52409f86407a9e76ab28a6db66f14e20ed7c84d3ea8aae2c97"} Oct 30 21:16:53.736659 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:53.736522 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="605c0491eda5ad52409f86407a9e76ab28a6db66f14e20ed7c84d3ea8aae2c97" Oct 30 21:16:53.927298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:53.927266 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:16:53.927443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:53.927340 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:16:54.755260 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:54.755206 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" event={"ID":"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc","Type":"ContainerStarted","Data":"c738ef7e1f84497e196df302f943f63912a80170c5238e14c045dc070953f6c2"} Oct 30 21:16:54.755994 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:54.755968 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:16:54.833056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:54.827253 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" podStartSLOduration=7.683788935 podStartE2EDuration="10.827203765s" totalImagesPullingTime="3.14341483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:16:54.812899569 +0000 UTC m=+1001.933973316" watchObservedRunningTime="2026-10-30 21:16:54.827203765 +0000 UTC m=+1001.948277513" Oct 30 21:16:55.765361 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:16:55.765331 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:17:03.808560 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:03.808525 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn"] Oct 30 21:17:03.809299 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:03.809282 4926 cpu_manager.go:414] "containerMap: removing container" podUID="8491486d-cb78-40a4-ab36-7ea5ece8a5a0" containerName="revoke-keys" Oct 30 21:17:03.809358 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:03.809301 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8491486d-cb78-40a4-ab36-7ea5ece8a5a0" containerName="revoke-keys" Oct 30 21:17:03.809405 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:03.809395 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="8491486d-cb78-40a4-ab36-7ea5ece8a5a0" containerName="revoke-keys" Oct 30 21:17:03.813449 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:03.813421 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn" Oct 30 21:17:03.831741 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:03.831711 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn"] Oct 30 21:17:03.839072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:03.839003 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:17:03.839411 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:03.839322 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:17:03.893104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:03.893062 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj27z\" (UniqueName: \"kubernetes.io/projected/7e56fc2e-931f-4896-a697-e20d2ffb4d76-kube-api-access-dj27z\") pod \"maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn\" (UID: \"7e56fc2e-931f-4896-a697-e20d2ffb4d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn" Oct 30 21:17:03.893388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:03.893368 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e56fc2e-931f-4896-a697-e20d2ffb4d76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn\" (UID: \"7e56fc2e-931f-4896-a697-e20d2ffb4d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn" Oct 30 21:17:03.893624 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:03.893586 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e56fc2e-931f-4896-a697-e20d2ffb4d76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn\" (UID: \"7e56fc2e-931f-4896-a697-e20d2ffb4d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn" Oct 30 21:17:03.994908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:03.994871 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj27z\" (UniqueName: \"kubernetes.io/projected/7e56fc2e-931f-4896-a697-e20d2ffb4d76-kube-api-access-dj27z\") pod \"maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn\" (UID: \"7e56fc2e-931f-4896-a697-e20d2ffb4d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn" Oct 30 21:17:03.995167 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:03.994964 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e56fc2e-931f-4896-a697-e20d2ffb4d76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn\" (UID: \"7e56fc2e-931f-4896-a697-e20d2ffb4d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn" Oct 30 21:17:03.995167 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:03.995122 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e56fc2e-931f-4896-a697-e20d2ffb4d76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn\" (UID: \"7e56fc2e-931f-4896-a697-e20d2ffb4d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn" Oct 30 21:17:03.995640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:03.995620 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e56fc2e-931f-4896-a697-e20d2ffb4d76-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn\" (UID: \"7e56fc2e-931f-4896-a697-e20d2ffb4d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn" Oct 30 21:17:03.997669 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:03.997644 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e56fc2e-931f-4896-a697-e20d2ffb4d76-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn\" (UID: \"7e56fc2e-931f-4896-a697-e20d2ffb4d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn" Oct 30 21:17:04.013493 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:04.013459 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj27z\" (UniqueName: \"kubernetes.io/projected/7e56fc2e-931f-4896-a697-e20d2ffb4d76-kube-api-access-dj27z\") pod \"maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn\" (UID: \"7e56fc2e-931f-4896-a697-e20d2ffb4d76\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn" Oct 30 21:17:04.129650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:04.129567 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn" Oct 30 21:17:04.437374 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:17:04.437338 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e56fc2e_931f_4896_a697_e20d2ffb4d76.slice/crio-b60690c8fa87c310c357bc7f45570d3710dcb7cbe0767ff90a01fba89ac31418 WatchSource:0}: Error finding container b60690c8fa87c310c357bc7f45570d3710dcb7cbe0767ff90a01fba89ac31418: Status 404 returned error can't find the container with id b60690c8fa87c310c357bc7f45570d3710dcb7cbe0767ff90a01fba89ac31418 Oct 30 21:17:04.438942 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:04.438868 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn"] Oct 30 21:17:04.849225 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:04.849193 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn" event={"ID":"7e56fc2e-931f-4896-a697-e20d2ffb4d76","Type":"ContainerStarted","Data":"b60690c8fa87c310c357bc7f45570d3710dcb7cbe0767ff90a01fba89ac31418"} Oct 30 21:17:05.860316 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:05.860231 4926 generic.go:356] "Generic (PLEG): container finished" podID="7e56fc2e-931f-4896-a697-e20d2ffb4d76" containerID="fdbf942d59abb497bdfd50563a99825deb465879cfc45bb3461926743132aa9d" exitCode=0 Oct 30 21:17:05.860689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:05.860309 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn" event={"ID":"7e56fc2e-931f-4896-a697-e20d2ffb4d76","Type":"ContainerDied","Data":"fdbf942d59abb497bdfd50563a99825deb465879cfc45bb3461926743132aa9d"} Oct 30 21:17:07.113295 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:07.113269 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn" Oct 30 21:17:07.252172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:07.252132 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e56fc2e-931f-4896-a697-e20d2ffb4d76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e56fc2e-931f-4896-a697-e20d2ffb4d76-maas-api-openshift-ca\") pod \"7e56fc2e-931f-4896-a697-e20d2ffb4d76\" (UID: \"7e56fc2e-931f-4896-a697-e20d2ffb4d76\") " Oct 30 21:17:07.252458 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:07.252425 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e56fc2e-931f-4896-a697-e20d2ffb4d76-kube-api-access-dj27z\" (UniqueName: \"kubernetes.io/projected/7e56fc2e-931f-4896-a697-e20d2ffb4d76-kube-api-access-dj27z\") pod \"7e56fc2e-931f-4896-a697-e20d2ffb4d76\" (UID: \"7e56fc2e-931f-4896-a697-e20d2ffb4d76\") " Oct 30 21:17:07.252563 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:07.252547 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e56fc2e-931f-4896-a697-e20d2ffb4d76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e56fc2e-931f-4896-a697-e20d2ffb4d76-maas-api-xks-ca\") pod \"7e56fc2e-931f-4896-a697-e20d2ffb4d76\" (UID: \"7e56fc2e-931f-4896-a697-e20d2ffb4d76\") " Oct 30 21:17:07.252626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:07.252562 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e56fc2e-931f-4896-a697-e20d2ffb4d76-maas-api-openshift-ca" pod "7e56fc2e-931f-4896-a697-e20d2ffb4d76" (UID: "7e56fc2e-931f-4896-a697-e20d2ffb4d76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:17:07.253152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:07.253128 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e56fc2e-931f-4896-a697-e20d2ffb4d76-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:17:07.254583 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:07.254553 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e56fc2e-931f-4896-a697-e20d2ffb4d76-kube-api-access-dj27z" pod "7e56fc2e-931f-4896-a697-e20d2ffb4d76" (UID: "7e56fc2e-931f-4896-a697-e20d2ffb4d76"). InnerVolumeSpecName "kube-api-access-dj27z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:07.254716 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:07.254698 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e56fc2e-931f-4896-a697-e20d2ffb4d76-maas-api-xks-ca" pod "7e56fc2e-931f-4896-a697-e20d2ffb4d76" (UID: "7e56fc2e-931f-4896-a697-e20d2ffb4d76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:17:07.354808 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:07.354768 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj27z\" (UniqueName: \"kubernetes.io/projected/7e56fc2e-931f-4896-a697-e20d2ffb4d76-kube-api-access-dj27z\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:17:07.354808 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:07.354802 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e56fc2e-931f-4896-a697-e20d2ffb4d76-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:17:07.488385 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:17:07.488351 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:17:07.881678 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:07.881650 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn" Oct 30 21:17:07.882012 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:07.881682 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn" event={"ID":"7e56fc2e-931f-4896-a697-e20d2ffb4d76","Type":"ContainerDied","Data":"b60690c8fa87c310c357bc7f45570d3710dcb7cbe0767ff90a01fba89ac31418"} Oct 30 21:17:07.882184 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:07.882064 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b60690c8fa87c310c357bc7f45570d3710dcb7cbe0767ff90a01fba89ac31418" Oct 30 21:17:09.166127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:09.166095 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx"] Oct 30 21:17:09.167007 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:09.166985 4926 cpu_manager.go:414] "containerMap: removing container" podUID="7e56fc2e-931f-4896-a697-e20d2ffb4d76" containerName="revoke-keys" Oct 30 21:17:09.167007 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:09.167007 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e56fc2e-931f-4896-a697-e20d2ffb4d76" containerName="revoke-keys" Oct 30 21:17:09.167216 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:09.167154 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e56fc2e-931f-4896-a697-e20d2ffb4d76" containerName="revoke-keys" Oct 30 21:17:09.171176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:09.171155 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx" Oct 30 21:17:09.188347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:09.188323 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx"] Oct 30 21:17:09.191812 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:09.191526 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:17:09.191812 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:09.191566 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:17:09.287917 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:09.287880 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f1125e3-ebee-49e7-8ead-5a716841c9a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx\" (UID: \"7f1125e3-ebee-49e7-8ead-5a716841c9a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx" Oct 30 21:17:09.288147 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:09.288087 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g5nw\" (UniqueName: \"kubernetes.io/projected/7f1125e3-ebee-49e7-8ead-5a716841c9a0-kube-api-access-2g5nw\") pod \"maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx\" (UID: \"7f1125e3-ebee-49e7-8ead-5a716841c9a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx" Oct 30 21:17:09.288228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:09.288210 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f1125e3-ebee-49e7-8ead-5a716841c9a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx\" (UID: \"7f1125e3-ebee-49e7-8ead-5a716841c9a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx" Oct 30 21:17:09.389829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:09.389780 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f1125e3-ebee-49e7-8ead-5a716841c9a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx\" (UID: \"7f1125e3-ebee-49e7-8ead-5a716841c9a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx" Oct 30 21:17:09.390060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:09.389935 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g5nw\" (UniqueName: \"kubernetes.io/projected/7f1125e3-ebee-49e7-8ead-5a716841c9a0-kube-api-access-2g5nw\") pod \"maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx\" (UID: \"7f1125e3-ebee-49e7-8ead-5a716841c9a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx" Oct 30 21:17:09.390110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:09.390062 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f1125e3-ebee-49e7-8ead-5a716841c9a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx\" (UID: \"7f1125e3-ebee-49e7-8ead-5a716841c9a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx" Oct 30 21:17:09.390649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:09.390619 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f1125e3-ebee-49e7-8ead-5a716841c9a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx\" (UID: \"7f1125e3-ebee-49e7-8ead-5a716841c9a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx" Oct 30 21:17:09.392705 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:09.392679 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f1125e3-ebee-49e7-8ead-5a716841c9a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx\" (UID: \"7f1125e3-ebee-49e7-8ead-5a716841c9a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx" Oct 30 21:17:09.408903 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:09.408869 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g5nw\" (UniqueName: \"kubernetes.io/projected/7f1125e3-ebee-49e7-8ead-5a716841c9a0-kube-api-access-2g5nw\") pod \"maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx\" (UID: \"7f1125e3-ebee-49e7-8ead-5a716841c9a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx" Oct 30 21:17:09.489161 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:09.489081 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx" Oct 30 21:17:09.805892 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:17:09.805850 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f1125e3_ebee_49e7_8ead_5a716841c9a0.slice/crio-27916e852b84f0f997114a02d14da1085a1c1c7548b008c6b00a7e8f5cc8fc5b WatchSource:0}: Error finding container 27916e852b84f0f997114a02d14da1085a1c1c7548b008c6b00a7e8f5cc8fc5b: Status 404 returned error can't find the container with id 27916e852b84f0f997114a02d14da1085a1c1c7548b008c6b00a7e8f5cc8fc5b Oct 30 21:17:09.806214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:09.806192 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx"] Oct 30 21:17:09.902567 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:09.902531 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx" event={"ID":"7f1125e3-ebee-49e7-8ead-5a716841c9a0","Type":"ContainerStarted","Data":"27916e852b84f0f997114a02d14da1085a1c1c7548b008c6b00a7e8f5cc8fc5b"} Oct 30 21:17:10.922075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:10.921598 4926 generic.go:356] "Generic (PLEG): container finished" podID="7f1125e3-ebee-49e7-8ead-5a716841c9a0" containerID="9b435b6e308da961b2acf352830265529a0cbbeeb043ec93a728dcafb8903d01" exitCode=0 Oct 30 21:17:10.922075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:10.921808 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx" event={"ID":"7f1125e3-ebee-49e7-8ead-5a716841c9a0","Type":"ContainerDied","Data":"9b435b6e308da961b2acf352830265529a0cbbeeb043ec93a728dcafb8903d01"} Oct 30 21:17:12.171629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:12.171598 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx" Oct 30 21:17:12.260310 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:12.260214 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f1125e3-ebee-49e7-8ead-5a716841c9a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f1125e3-ebee-49e7-8ead-5a716841c9a0-maas-api-openshift-ca\") pod \"7f1125e3-ebee-49e7-8ead-5a716841c9a0\" (UID: \"7f1125e3-ebee-49e7-8ead-5a716841c9a0\") " Oct 30 21:17:12.260490 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:12.260359 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f1125e3-ebee-49e7-8ead-5a716841c9a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f1125e3-ebee-49e7-8ead-5a716841c9a0-maas-api-xks-ca\") pod \"7f1125e3-ebee-49e7-8ead-5a716841c9a0\" (UID: \"7f1125e3-ebee-49e7-8ead-5a716841c9a0\") " Oct 30 21:17:12.260490 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:12.260399 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f1125e3-ebee-49e7-8ead-5a716841c9a0-kube-api-access-2g5nw\" (UniqueName: \"kubernetes.io/projected/7f1125e3-ebee-49e7-8ead-5a716841c9a0-kube-api-access-2g5nw\") pod \"7f1125e3-ebee-49e7-8ead-5a716841c9a0\" (UID: \"7f1125e3-ebee-49e7-8ead-5a716841c9a0\") " Oct 30 21:17:12.260673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:12.260644 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f1125e3-ebee-49e7-8ead-5a716841c9a0-maas-api-openshift-ca" pod "7f1125e3-ebee-49e7-8ead-5a716841c9a0" (UID: "7f1125e3-ebee-49e7-8ead-5a716841c9a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:17:12.261375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:12.261347 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f1125e3-ebee-49e7-8ead-5a716841c9a0-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:17:12.263439 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:12.263412 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f1125e3-ebee-49e7-8ead-5a716841c9a0-maas-api-xks-ca" pod "7f1125e3-ebee-49e7-8ead-5a716841c9a0" (UID: "7f1125e3-ebee-49e7-8ead-5a716841c9a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:17:12.264839 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:12.264803 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f1125e3-ebee-49e7-8ead-5a716841c9a0-kube-api-access-2g5nw" pod "7f1125e3-ebee-49e7-8ead-5a716841c9a0" (UID: "7f1125e3-ebee-49e7-8ead-5a716841c9a0"). InnerVolumeSpecName "kube-api-access-2g5nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:12.362776 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:12.362745 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f1125e3-ebee-49e7-8ead-5a716841c9a0-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:17:12.362776 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:12.362772 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g5nw\" (UniqueName: \"kubernetes.io/projected/7f1125e3-ebee-49e7-8ead-5a716841c9a0-kube-api-access-2g5nw\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:17:12.944260 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:12.944227 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx" Oct 30 21:17:12.944260 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:12.944244 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx" event={"ID":"7f1125e3-ebee-49e7-8ead-5a716841c9a0","Type":"ContainerDied","Data":"27916e852b84f0f997114a02d14da1085a1c1c7548b008c6b00a7e8f5cc8fc5b"} Oct 30 21:17:12.944472 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:12.944293 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27916e852b84f0f997114a02d14da1085a1c1c7548b008c6b00a7e8f5cc8fc5b" Oct 30 21:17:17.625644 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:17.625589 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:17:17.626650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:17.626249 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:17:17.893120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:17.882138 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:17:18.046065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:18.043137 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl"] Oct 30 21:17:18.068056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:18.052717 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97367ea0-v5wdl"] Oct 30 21:17:19.123057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:19.122517 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3d6e1d2-c720-4883-a300-501d5b3847b7" path="/var/lib/kubelet/pods/c3d6e1d2-c720-4883-a300-501d5b3847b7/volumes" Oct 30 21:17:23.170168 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:23.170099 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn"] Oct 30 21:17:23.180946 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:23.180917 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-dd1e164d-hz4rn"] Oct 30 21:17:25.019072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:25.018972 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e56fc2e-931f-4896-a697-e20d2ffb4d76" path="/var/lib/kubelet/pods/7e56fc2e-931f-4896-a697-e20d2ffb4d76/volumes" Oct 30 21:17:30.304835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:30.304798 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp"] Oct 30 21:17:30.305672 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:30.305620 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" podUID="4820b8d6-9171-4e76-a572-7e8d2007280c" containerName="maas-api" containerID="cri-o://53e2db74f0d08fe62f8dab8f7bfc5ead2cf2addbc179051ac8e279eab434542c" gracePeriod=30 Oct 30 21:17:30.748583 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:30.748541 4926 generic.go:356] "Generic (PLEG): container finished" podID="4820b8d6-9171-4e76-a572-7e8d2007280c" containerID="53e2db74f0d08fe62f8dab8f7bfc5ead2cf2addbc179051ac8e279eab434542c" exitCode=0 Oct 30 21:17:30.748957 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:30.748926 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" event={"ID":"4820b8d6-9171-4e76-a572-7e8d2007280c","Type":"ContainerDied","Data":"53e2db74f0d08fe62f8dab8f7bfc5ead2cf2addbc179051ac8e279eab434542c"} Oct 30 21:17:30.784048 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:30.784005 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" Oct 30 21:17:30.991054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:30.990710 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4820b8d6-9171-4e76-a572-7e8d2007280c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4820b8d6-9171-4e76-a572-7e8d2007280c-maas-api-tls\") pod \"4820b8d6-9171-4e76-a572-7e8d2007280c\" (UID: \"4820b8d6-9171-4e76-a572-7e8d2007280c\") " Oct 30 21:17:30.991054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:30.990963 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4820b8d6-9171-4e76-a572-7e8d2007280c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4820b8d6-9171-4e76-a572-7e8d2007280c-metrics-certs\") pod \"4820b8d6-9171-4e76-a572-7e8d2007280c\" (UID: \"4820b8d6-9171-4e76-a572-7e8d2007280c\") " Oct 30 21:17:30.991054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:30.991010 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4820b8d6-9171-4e76-a572-7e8d2007280c-kube-api-access-6dvcf\" (UniqueName: \"kubernetes.io/projected/4820b8d6-9171-4e76-a572-7e8d2007280c-kube-api-access-6dvcf\") pod \"4820b8d6-9171-4e76-a572-7e8d2007280c\" (UID: \"4820b8d6-9171-4e76-a572-7e8d2007280c\") " Oct 30 21:17:30.991368 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:30.991099 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4820b8d6-9171-4e76-a572-7e8d2007280c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4820b8d6-9171-4e76-a572-7e8d2007280c-openshift-service-ca\") pod \"4820b8d6-9171-4e76-a572-7e8d2007280c\" (UID: \"4820b8d6-9171-4e76-a572-7e8d2007280c\") " Oct 30 21:17:31.004142 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:30.992399 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4820b8d6-9171-4e76-a572-7e8d2007280c-openshift-service-ca" pod "4820b8d6-9171-4e76-a572-7e8d2007280c" (UID: "4820b8d6-9171-4e76-a572-7e8d2007280c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:17:31.008264 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:31.007942 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4820b8d6-9171-4e76-a572-7e8d2007280c-maas-api-tls" pod "4820b8d6-9171-4e76-a572-7e8d2007280c" (UID: "4820b8d6-9171-4e76-a572-7e8d2007280c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:17:31.110057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:31.100978 4926 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4820b8d6-9171-4e76-a572-7e8d2007280c-openshift-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:17:31.110057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:31.101011 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4820b8d6-9171-4e76-a572-7e8d2007280c-maas-api-tls\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:17:31.126057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:31.125598 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04757449-2s5fd"] Oct 30 21:17:31.142597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:31.142510 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4820b8d6-9171-4e76-a572-7e8d2007280c-metrics-certs" pod "4820b8d6-9171-4e76-a572-7e8d2007280c" (UID: "4820b8d6-9171-4e76-a572-7e8d2007280c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:17:31.148908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:31.148879 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04757449-2s5fd"] Oct 30 21:17:31.149679 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:31.149651 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4820b8d6-9171-4e76-a572-7e8d2007280c-kube-api-access-6dvcf" pod "4820b8d6-9171-4e76-a572-7e8d2007280c" (UID: "4820b8d6-9171-4e76-a572-7e8d2007280c"). InnerVolumeSpecName "kube-api-access-6dvcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:17:31.203088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:31.202650 4926 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4820b8d6-9171-4e76-a572-7e8d2007280c-metrics-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:17:31.203088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:31.202683 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dvcf\" (UniqueName: \"kubernetes.io/projected/4820b8d6-9171-4e76-a572-7e8d2007280c-kube-api-access-6dvcf\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:17:31.690717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:31.690681 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx"] Oct 30 21:17:31.709256 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:31.704798 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4b62455c-hm8mx"] Oct 30 21:17:31.764158 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:31.764131 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" Oct 30 21:17:31.764384 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:31.764264 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp" event={"ID":"4820b8d6-9171-4e76-a572-7e8d2007280c","Type":"ContainerDied","Data":"7ac1586fa94fc3df7a6819b02f9d451a8cd35ef7902a619dabcad8cd10b59e04"} Oct 30 21:17:31.764384 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:31.764325 4926 scope.go:122] "RemoveContainer" containerID="53e2db74f0d08fe62f8dab8f7bfc5ead2cf2addbc179051ac8e279eab434542c" Oct 30 21:17:31.848961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:31.834219 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp"] Oct 30 21:17:31.873104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:31.872753 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4b62455c-794d66845b-4wfvp"] Oct 30 21:17:33.051537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:33.050832 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4820b8d6-9171-4e76-a572-7e8d2007280c" path="/var/lib/kubelet/pods/4820b8d6-9171-4e76-a572-7e8d2007280c/volumes" Oct 30 21:17:33.055622 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:33.053101 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f1125e3-ebee-49e7-8ead-5a716841c9a0" path="/var/lib/kubelet/pods/7f1125e3-ebee-49e7-8ead-5a716841c9a0/volumes" Oct 30 21:17:33.055622 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:33.053721 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b3c1f7-f3b0-4f7b-8359-96002c9840fb" path="/var/lib/kubelet/pods/84b3c1f7-f3b0-4f7b-8359-96002c9840fb/volumes" Oct 30 21:17:33.068064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:33.064474 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-660e8d66-l5s9r"] Oct 30 21:17:33.078098 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:33.074599 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-660e8d66-l5s9r"] Oct 30 21:17:35.033501 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:35.033466 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f419ca1-8aa5-449e-bde2-1d3c6de70c84" path="/var/lib/kubelet/pods/6f419ca1-8aa5-449e-bde2-1d3c6de70c84/volumes" Oct 30 21:17:37.036345 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:37.036312 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth"] Oct 30 21:17:37.043180 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:37.043150 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ef521603-xmvth"] Oct 30 21:17:37.492650 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:17:37.492607 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:17:39.017689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:39.017657 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2d0afaf-bbfd-4383-858a-f5d78c46032a" path="/var/lib/kubelet/pods/b2d0afaf-bbfd-4383-858a-f5d78c46032a/volumes" Oct 30 21:17:39.220197 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:39.220142 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:17:42.846055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:42.845123 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:17:43.039248 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:43.039202 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:17:43.068051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:43.066562 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:17:43.088052 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:43.075785 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:17:43.249172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:43.249122 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:17:43.288283 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:43.288237 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 21:17:43.543225 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:43.543118 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:17:43.558331 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:43.558290 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 21:17:43.570501 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:43.570457 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:17:43.592581 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:43.592532 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:17:44.039529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:44.039489 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e8c81a-qgb9p"] Oct 30 21:17:44.046458 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:44.046433 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e8c81a-qgb9p"] Oct 30 21:17:45.021140 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:45.021103 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fd710cc-405e-4c11-82b9-82e3d8ec793b" path="/var/lib/kubelet/pods/0fd710cc-405e-4c11-82b9-82e3d8ec793b/volumes" Oct 30 21:17:51.034655 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:51.034617 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-842e1acf-mcm4q"] Oct 30 21:17:51.044266 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:51.044237 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-842e1acf-mcm4q"] Oct 30 21:17:51.094882 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:51.094850 4926 generic.go:356] "Generic (PLEG): container finished" podID="3f4943da-3bdc-4e2c-900a-5c286a52bd09" containerID="d515d4e57a4ce92641851cdda30fd632453315210af5f0c51d2860e068d38550" exitCode=0 Oct 30 21:17:51.095088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:51.094925 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" event={"ID":"3f4943da-3bdc-4e2c-900a-5c286a52bd09","Type":"ContainerDied","Data":"d515d4e57a4ce92641851cdda30fd632453315210af5f0c51d2860e068d38550"} Oct 30 21:17:51.095465 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:51.095448 4926 scope.go:122] "RemoveContainer" containerID="d515d4e57a4ce92641851cdda30fd632453315210af5f0c51d2860e068d38550" Oct 30 21:17:51.097379 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:51.097348 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:17:51.097463 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:51.097434 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:17:51.097502 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:51.097480 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:17:52.110972 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:52.110937 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" event={"ID":"3f4943da-3bdc-4e2c-900a-5c286a52bd09","Type":"ContainerStarted","Data":"eb407a28aabe5af1cc008df682fd42330d4d9a520f982e2ca159539beed29ac4"} Oct 30 21:17:52.991058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:52.990448 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:17:52.991058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:52.990585 4926 patch_prober.go:29] interesting pod/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:17:52.991058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:52.990628 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" podUID="3f4943da-3bdc-4e2c-900a-5c286a52bd09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 21:17:53.022136 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:53.019845 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5a39e44-a5b5-46be-a81a-4d9b8c840037" path="/var/lib/kubelet/pods/f5a39e44-a5b5-46be-a81a-4d9b8c840037/volumes" Oct 30 21:17:53.996486 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:53.996443 4926 patch_prober.go:29] interesting pod/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:17:53.997223 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:53.997188 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" podUID="3f4943da-3bdc-4e2c-900a-5c286a52bd09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 21:17:55.000128 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:54.988384 4926 patch_prober.go:29] interesting pod/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:17:55.000128 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:54.988447 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" podUID="3f4943da-3bdc-4e2c-900a-5c286a52bd09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 21:17:55.988399 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:55.988364 4926 patch_prober.go:29] interesting pod/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:17:55.988722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:55.988672 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" podUID="3f4943da-3bdc-4e2c-900a-5c286a52bd09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 21:17:56.992057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:56.991125 4926 patch_prober.go:29] interesting pod/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 21:17:56.992057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:56.991199 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" podUID="3f4943da-3bdc-4e2c-900a-5c286a52bd09" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 21:17:57.990225 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:57.990194 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:17:57.990757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:57.990729 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:17:57.991529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:17:57.991511 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:18:07.486398 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:18:07.486359 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:18:08.042234 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:08.042201 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a3cb196-mb2f2"] Oct 30 21:18:08.048551 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:08.048516 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a3cb196-mb2f2"] Oct 30 21:18:09.018161 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:09.018130 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a329ae7-103e-483a-9e33-94276f6f4d72" path="/var/lib/kubelet/pods/1a329ae7-103e-483a-9e33-94276f6f4d72/volumes" Oct 30 21:18:15.079121 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:15.079080 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5"] Oct 30 21:18:15.080182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:15.080159 4926 cpu_manager.go:414] "containerMap: removing container" podUID="4820b8d6-9171-4e76-a572-7e8d2007280c" containerName="maas-api" Oct 30 21:18:15.080284 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:15.080184 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4820b8d6-9171-4e76-a572-7e8d2007280c" containerName="maas-api" Oct 30 21:18:15.080284 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:15.080231 4926 cpu_manager.go:414] "containerMap: removing container" podUID="7f1125e3-ebee-49e7-8ead-5a716841c9a0" containerName="revoke-keys" Oct 30 21:18:15.080284 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:15.080240 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f1125e3-ebee-49e7-8ead-5a716841c9a0" containerName="revoke-keys" Oct 30 21:18:15.080403 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:15.080392 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f1125e3-ebee-49e7-8ead-5a716841c9a0" containerName="revoke-keys" Oct 30 21:18:15.080438 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:15.080416 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="4820b8d6-9171-4e76-a572-7e8d2007280c" containerName="maas-api" Oct 30 21:18:15.086135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:15.086113 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5" Oct 30 21:18:15.099150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:15.099020 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5"] Oct 30 21:18:15.104714 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:15.104689 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:18:15.104831 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:15.104755 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:18:15.158679 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:15.158644 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc2e803-8410-4ef4-83d7-035cc04f9594-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c906b806-glpn5\" (UID: \"8cc2e803-8410-4ef4-83d7-035cc04f9594\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5" Oct 30 21:18:15.158925 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:15.158904 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc2e803-8410-4ef4-83d7-035cc04f9594-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c906b806-glpn5\" (UID: \"8cc2e803-8410-4ef4-83d7-035cc04f9594\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5" Oct 30 21:18:15.159054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:15.159011 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl8mq\" (UniqueName: \"kubernetes.io/projected/8cc2e803-8410-4ef4-83d7-035cc04f9594-kube-api-access-jl8mq\") pod \"maas-api-revoke-keys-e2e-derive-c906b806-glpn5\" (UID: \"8cc2e803-8410-4ef4-83d7-035cc04f9594\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5" Oct 30 21:18:15.260042 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:15.259999 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc2e803-8410-4ef4-83d7-035cc04f9594-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c906b806-glpn5\" (UID: \"8cc2e803-8410-4ef4-83d7-035cc04f9594\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5" Oct 30 21:18:15.260265 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:15.260246 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc2e803-8410-4ef4-83d7-035cc04f9594-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c906b806-glpn5\" (UID: \"8cc2e803-8410-4ef4-83d7-035cc04f9594\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5" Oct 30 21:18:15.260333 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:15.260309 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl8mq\" (UniqueName: \"kubernetes.io/projected/8cc2e803-8410-4ef4-83d7-035cc04f9594-kube-api-access-jl8mq\") pod \"maas-api-revoke-keys-e2e-derive-c906b806-glpn5\" (UID: \"8cc2e803-8410-4ef4-83d7-035cc04f9594\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5" Oct 30 21:18:15.260933 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:15.260906 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc2e803-8410-4ef4-83d7-035cc04f9594-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-c906b806-glpn5\" (UID: \"8cc2e803-8410-4ef4-83d7-035cc04f9594\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5" Oct 30 21:18:15.262343 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:15.262326 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc2e803-8410-4ef4-83d7-035cc04f9594-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-c906b806-glpn5\" (UID: \"8cc2e803-8410-4ef4-83d7-035cc04f9594\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5" Oct 30 21:18:15.281719 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:15.281689 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl8mq\" (UniqueName: \"kubernetes.io/projected/8cc2e803-8410-4ef4-83d7-035cc04f9594-kube-api-access-jl8mq\") pod \"maas-api-revoke-keys-e2e-derive-c906b806-glpn5\" (UID: \"8cc2e803-8410-4ef4-83d7-035cc04f9594\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5" Oct 30 21:18:15.407675 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:15.407586 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5" Oct 30 21:18:15.717405 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:15.717378 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5"] Oct 30 21:18:15.718196 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:18:15.718167 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc2e803_8410_4ef4_83d7_035cc04f9594.slice/crio-55222a15968ee410a370383fb0e9c4e8af97a48fe30eadf87796f6a3f382911a WatchSource:0}: Error finding container 55222a15968ee410a370383fb0e9c4e8af97a48fe30eadf87796f6a3f382911a: Status 404 returned error can't find the container with id 55222a15968ee410a370383fb0e9c4e8af97a48fe30eadf87796f6a3f382911a Oct 30 21:18:16.448884 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:16.448846 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5" event={"ID":"8cc2e803-8410-4ef4-83d7-035cc04f9594","Type":"ContainerStarted","Data":"af4d55d33e63f8e49c87237ab6576a9282d90a72e8a090171724a2e4dd3fd04b"} Oct 30 21:18:16.449308 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:16.448943 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5" event={"ID":"8cc2e803-8410-4ef4-83d7-035cc04f9594","Type":"ContainerStarted","Data":"55222a15968ee410a370383fb0e9c4e8af97a48fe30eadf87796f6a3f382911a"} Oct 30 21:18:16.496589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:16.496514 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5" podStartSLOduration=1.496495605 podStartE2EDuration="1.496495605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:18:16.489570754 +0000 UTC m=+1083.610644501" watchObservedRunningTime="2026-10-30 21:18:16.496495605 +0000 UTC m=+1083.617569350" Oct 30 21:18:17.077505 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:17.077426 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:18:17.461020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:17.460978 4926 generic.go:356] "Generic (PLEG): container finished" podID="8cc2e803-8410-4ef4-83d7-035cc04f9594" containerID="af4d55d33e63f8e49c87237ab6576a9282d90a72e8a090171724a2e4dd3fd04b" exitCode=0 Oct 30 21:18:17.461574 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:17.461057 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5" event={"ID":"8cc2e803-8410-4ef4-83d7-035cc04f9594","Type":"ContainerDied","Data":"af4d55d33e63f8e49c87237ab6576a9282d90a72e8a090171724a2e4dd3fd04b"} Oct 30 21:18:18.881068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:18.881011 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5" Oct 30 21:18:18.972660 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:18.972620 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cc2e803-8410-4ef4-83d7-035cc04f9594-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc2e803-8410-4ef4-83d7-035cc04f9594-maas-api-xks-ca\") pod \"8cc2e803-8410-4ef4-83d7-035cc04f9594\" (UID: \"8cc2e803-8410-4ef4-83d7-035cc04f9594\") " Oct 30 21:18:18.973226 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:18.973195 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cc2e803-8410-4ef4-83d7-035cc04f9594-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc2e803-8410-4ef4-83d7-035cc04f9594-maas-api-openshift-ca\") pod \"8cc2e803-8410-4ef4-83d7-035cc04f9594\" (UID: \"8cc2e803-8410-4ef4-83d7-035cc04f9594\") " Oct 30 21:18:18.973471 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:18.973453 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cc2e803-8410-4ef4-83d7-035cc04f9594-kube-api-access-jl8mq\" (UniqueName: \"kubernetes.io/projected/8cc2e803-8410-4ef4-83d7-035cc04f9594-kube-api-access-jl8mq\") pod \"8cc2e803-8410-4ef4-83d7-035cc04f9594\" (UID: \"8cc2e803-8410-4ef4-83d7-035cc04f9594\") " Oct 30 21:18:18.973641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:18.973587 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cc2e803-8410-4ef4-83d7-035cc04f9594-maas-api-openshift-ca" pod "8cc2e803-8410-4ef4-83d7-035cc04f9594" (UID: "8cc2e803-8410-4ef4-83d7-035cc04f9594"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:18:18.974367 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:18.974274 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc2e803-8410-4ef4-83d7-035cc04f9594-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:18:18.981744 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:18.980969 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cc2e803-8410-4ef4-83d7-035cc04f9594-kube-api-access-jl8mq" pod "8cc2e803-8410-4ef4-83d7-035cc04f9594" (UID: "8cc2e803-8410-4ef4-83d7-035cc04f9594"). InnerVolumeSpecName "kube-api-access-jl8mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:18:18.988767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:18.988296 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cc2e803-8410-4ef4-83d7-035cc04f9594-maas-api-xks-ca" pod "8cc2e803-8410-4ef4-83d7-035cc04f9594" (UID: "8cc2e803-8410-4ef4-83d7-035cc04f9594"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:18:19.076639 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:19.076599 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl8mq\" (UniqueName: \"kubernetes.io/projected/8cc2e803-8410-4ef4-83d7-035cc04f9594-kube-api-access-jl8mq\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:18:19.076639 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:19.076640 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc2e803-8410-4ef4-83d7-035cc04f9594-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:18:19.506315 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:19.506270 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5" Oct 30 21:18:19.506520 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:19.506275 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5" event={"ID":"8cc2e803-8410-4ef4-83d7-035cc04f9594","Type":"ContainerDied","Data":"55222a15968ee410a370383fb0e9c4e8af97a48fe30eadf87796f6a3f382911a"} Oct 30 21:18:19.506520 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:19.506441 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55222a15968ee410a370383fb0e9c4e8af97a48fe30eadf87796f6a3f382911a" Oct 30 21:18:28.866510 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:28.866475 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5"] Oct 30 21:18:28.879508 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:28.879478 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c906b806-glpn5"] Oct 30 21:18:29.033106 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:18:29.033070 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cc2e803-8410-4ef4-83d7-035cc04f9594" path="/var/lib/kubelet/pods/8cc2e803-8410-4ef4-83d7-035cc04f9594/volumes" Oct 30 21:18:37.488605 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:18:37.488559 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:19:07.486891 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:19:07.486849 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:19:08.065723 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:08.065684 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:19:12.277165 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:12.277118 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 21:19:12.307218 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:12.307178 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cdm7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:19:12.307521 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:12.307487 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" podUID="9cbcfa4c-ad1b-4110-a069-38c79809bc98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:19:13.330514 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:13.330474 4926 scope.go:122] "RemoveContainer" containerID="5f9ce042e4939ff63cd6725a4f5e6991e94b2335553a764aa33cb4b019b0e72a" Oct 30 21:19:13.348227 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:13.348188 4926 scope.go:122] "RemoveContainer" containerID="97d0a7567d2e36c866d3d074db67e3281df1f0261e88f098daed2e27219d23c4" Oct 30 21:19:13.360296 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:13.360217 4926 scope.go:122] "RemoveContainer" containerID="462d008cb14df15b2d27906660a78599b32cfc9fb5b22544c48ad26b3946d1ff" Oct 30 21:19:13.395589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:13.394816 4926 scope.go:122] "RemoveContainer" containerID="21744da6d7062052128964b868bcf6843baa375a0f87032e5bdd67341122a5b4" Oct 30 21:19:13.409250 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:13.409223 4926 scope.go:122] "RemoveContainer" containerID="4c96d6646d242fc8b6e397b29948355d6161ceacf05170033b096a5e159d703e" Oct 30 21:19:13.421925 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:13.421903 4926 scope.go:122] "RemoveContainer" containerID="f21b61e0c463b08366fd6c01f7b0e1645a0180405081a805b9b3f894d3ced0ac" Oct 30 21:19:13.432348 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:13.432325 4926 scope.go:122] "RemoveContainer" containerID="7e01242e5029a737877f4ba2728d5cd62340f92bf4faeee712a8e3994014ec07" Oct 30 21:19:37.484301 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:19:37.484258 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:19:38.062176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:38.061075 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6426881-gqhqk"] Oct 30 21:19:38.085097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:38.080131 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6426881-gqhqk"] Oct 30 21:19:39.018142 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:39.018097 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a54c7ae-6d2e-4afb-93af-7f712c7995b6" path="/var/lib/kubelet/pods/5a54c7ae-6d2e-4afb-93af-7f712c7995b6/volumes" Oct 30 21:19:45.645892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:45.645848 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm"] Oct 30 21:19:45.649386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:45.649351 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" podUID="3f4943da-3bdc-4e2c-900a-5c286a52bd09" containerName="istio-proxy" containerID="cri-o://eb407a28aabe5af1cc008df682fd42330d4d9a520f982e2ca159539beed29ac4" gracePeriod=30 Oct 30 21:19:50.987211 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:50.987183 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:19:51.034796 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.034766 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5"] Oct 30 21:19:51.039436 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.039411 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a7ae484-q7jn5"] Oct 30 21:19:51.097906 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.097873 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-data\") pod \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " Oct 30 21:19:51.098137 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.097943 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-workload-socket\") pod \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " Oct 30 21:19:51.098137 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.098026 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4943da-3bdc-4e2c-900a-5c286a52bd09-kube-api-access-82f7j\" (UniqueName: \"kubernetes.io/projected/3f4943da-3bdc-4e2c-900a-5c286a52bd09-kube-api-access-82f7j\") pod \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " Oct 30 21:19:51.098137 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.098107 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-token\" (UniqueName: \"kubernetes.io/projected/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-token\") pod \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " Oct 30 21:19:51.098370 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.098154 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istiod-ca-cert\") pod \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " Oct 30 21:19:51.098370 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.098218 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-data" pod "3f4943da-3bdc-4e2c-900a-5c286a52bd09" (UID: "3f4943da-3bdc-4e2c-900a-5c286a52bd09"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:51.098370 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.098231 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-credential-socket\") pod \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " Oct 30 21:19:51.098370 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.098328 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-workload-certs\") pod \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " Oct 30 21:19:51.098587 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.098438 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-envoy\") pod \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " Oct 30 21:19:51.098587 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.098541 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-credential-socket" pod "3f4943da-3bdc-4e2c-900a-5c286a52bd09" (UID: "3f4943da-3bdc-4e2c-900a-5c286a52bd09"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:51.098721 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.098704 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-podinfo\") pod \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\" (UID: \"3f4943da-3bdc-4e2c-900a-5c286a52bd09\") " Oct 30 21:19:51.098799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.098765 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-workload-socket" pod "3f4943da-3bdc-4e2c-900a-5c286a52bd09" (UID: "3f4943da-3bdc-4e2c-900a-5c286a52bd09"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:51.099462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.099440 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:19:51.099587 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.099466 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:19:51.099587 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.099479 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:19:51.100002 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.099958 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-workload-certs" pod "3f4943da-3bdc-4e2c-900a-5c286a52bd09" (UID: "3f4943da-3bdc-4e2c-900a-5c286a52bd09"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:51.100364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.100342 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istiod-ca-cert" pod "3f4943da-3bdc-4e2c-900a-5c286a52bd09" (UID: "3f4943da-3bdc-4e2c-900a-5c286a52bd09"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:19:51.101852 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.101822 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4943da-3bdc-4e2c-900a-5c286a52bd09-kube-api-access-82f7j" pod "3f4943da-3bdc-4e2c-900a-5c286a52bd09" (UID: "3f4943da-3bdc-4e2c-900a-5c286a52bd09"). InnerVolumeSpecName "kube-api-access-82f7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:51.102934 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.102525 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-envoy" pod "3f4943da-3bdc-4e2c-900a-5c286a52bd09" (UID: "3f4943da-3bdc-4e2c-900a-5c286a52bd09"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:19:51.104994 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.104954 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-podinfo" pod "3f4943da-3bdc-4e2c-900a-5c286a52bd09" (UID: "3f4943da-3bdc-4e2c-900a-5c286a52bd09"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:19:51.106459 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.106432 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-token" pod "3f4943da-3bdc-4e2c-900a-5c286a52bd09" (UID: "3f4943da-3bdc-4e2c-900a-5c286a52bd09"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:19:51.200903 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.200876 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82f7j\" (UniqueName: \"kubernetes.io/projected/3f4943da-3bdc-4e2c-900a-5c286a52bd09-kube-api-access-82f7j\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:19:51.200903 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.200905 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:19:51.201094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.200914 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:19:51.201094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.200924 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:19:51.201094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.200932 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:19:51.201094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.200939 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f4943da-3bdc-4e2c-900a-5c286a52bd09-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:19:51.668385 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.668350 4926 generic.go:356] "Generic (PLEG): container finished" podID="3f4943da-3bdc-4e2c-900a-5c286a52bd09" containerID="eb407a28aabe5af1cc008df682fd42330d4d9a520f982e2ca159539beed29ac4" exitCode=0 Oct 30 21:19:51.668560 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.668399 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" event={"ID":"3f4943da-3bdc-4e2c-900a-5c286a52bd09","Type":"ContainerDied","Data":"eb407a28aabe5af1cc008df682fd42330d4d9a520f982e2ca159539beed29ac4"} Oct 30 21:19:51.668560 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.668441 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" event={"ID":"3f4943da-3bdc-4e2c-900a-5c286a52bd09","Type":"ContainerDied","Data":"0472767544c1e187351a8794d763ccf4ff5bf60bfee3800c57cc656185399bac"} Oct 30 21:19:51.668560 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.668468 4926 scope.go:122] "RemoveContainer" containerID="eb407a28aabe5af1cc008df682fd42330d4d9a520f982e2ca159539beed29ac4" Oct 30 21:19:51.668560 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.668405 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm" Oct 30 21:19:51.682414 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.682392 4926 scope.go:122] "RemoveContainer" containerID="d515d4e57a4ce92641851cdda30fd632453315210af5f0c51d2860e068d38550" Oct 30 21:19:51.695210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.695188 4926 scope.go:122] "RemoveContainer" containerID="eb407a28aabe5af1cc008df682fd42330d4d9a520f982e2ca159539beed29ac4" Oct 30 21:19:51.695534 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:19:51.695511 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb407a28aabe5af1cc008df682fd42330d4d9a520f982e2ca159539beed29ac4\": container with ID starting with eb407a28aabe5af1cc008df682fd42330d4d9a520f982e2ca159539beed29ac4 not found: ID does not exist" containerID="eb407a28aabe5af1cc008df682fd42330d4d9a520f982e2ca159539beed29ac4" Oct 30 21:19:51.695610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.695545 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb407a28aabe5af1cc008df682fd42330d4d9a520f982e2ca159539beed29ac4"} err="failed to get container status \"eb407a28aabe5af1cc008df682fd42330d4d9a520f982e2ca159539beed29ac4\": rpc error: code = NotFound desc = could not find container \"eb407a28aabe5af1cc008df682fd42330d4d9a520f982e2ca159539beed29ac4\": container with ID starting with eb407a28aabe5af1cc008df682fd42330d4d9a520f982e2ca159539beed29ac4 not found: ID does not exist" Oct 30 21:19:51.695610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.695564 4926 scope.go:122] "RemoveContainer" containerID="d515d4e57a4ce92641851cdda30fd632453315210af5f0c51d2860e068d38550" Oct 30 21:19:51.695827 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:19:51.695808 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d515d4e57a4ce92641851cdda30fd632453315210af5f0c51d2860e068d38550\": container with ID starting with d515d4e57a4ce92641851cdda30fd632453315210af5f0c51d2860e068d38550 not found: ID does not exist" containerID="d515d4e57a4ce92641851cdda30fd632453315210af5f0c51d2860e068d38550" Oct 30 21:19:51.695890 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.695836 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d515d4e57a4ce92641851cdda30fd632453315210af5f0c51d2860e068d38550"} err="failed to get container status \"d515d4e57a4ce92641851cdda30fd632453315210af5f0c51d2860e068d38550\": rpc error: code = NotFound desc = could not find container \"d515d4e57a4ce92641851cdda30fd632453315210af5f0c51d2860e068d38550\": container with ID starting with d515d4e57a4ce92641851cdda30fd632453315210af5f0c51d2860e068d38550 not found: ID does not exist" Oct 30 21:19:51.732073 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.732006 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm"] Oct 30 21:19:51.738149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:51.738116 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1f954ca9-openshift-default-ffd455d85-hnrnm"] Oct 30 21:19:53.019678 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:53.019644 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f4943da-3bdc-4e2c-900a-5c286a52bd09" path="/var/lib/kubelet/pods/3f4943da-3bdc-4e2c-900a-5c286a52bd09/volumes" Oct 30 21:19:53.020235 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:19:53.020220 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="492f0633-33e2-4c68-8ac5-342fc9a84db8" path="/var/lib/kubelet/pods/492f0633-33e2-4c68-8ac5-342fc9a84db8/volumes" Oct 30 21:20:00.789056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:00.788839 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:20:07.485587 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:20:07.485549 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:20:09.047422 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:09.047383 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6"] Oct 30 21:20:09.055757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:09.055724 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9866b83-qwjf6"] Oct 30 21:20:11.017517 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:11.017483 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a61ce7-23f7-4719-8725-67972c9f6e88" path="/var/lib/kubelet/pods/84a61ce7-23f7-4719-8725-67972c9f6e88/volumes" Oct 30 21:20:13.269097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:13.269064 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:20:13.271021 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:13.270995 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:20:28.027920 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:28.027846 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a14ffe5-xtn68"] Oct 30 21:20:28.038370 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:28.038339 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a14ffe5-xtn68"] Oct 30 21:20:29.019446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:29.019410 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a780d05f-ec0c-4645-8388-07282de67878" path="/var/lib/kubelet/pods/a780d05f-ec0c-4645-8388-07282de67878/volumes" Oct 30 21:20:31.085971 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:31.085938 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf"] Oct 30 21:20:31.103084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:31.103049 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0556107e-j7shf"] Oct 30 21:20:33.017877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:33.017836 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4" path="/var/lib/kubelet/pods/d70aee08-a8a7-4ef5-95b6-1ea7f0f0f1e4/volumes" Oct 30 21:20:34.631386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:34.631349 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfae3730-snd5s"] Oct 30 21:20:34.632245 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:34.632223 4926 cpu_manager.go:414] "containerMap: removing container" podUID="3f4943da-3bdc-4e2c-900a-5c286a52bd09" containerName="istio-proxy" Oct 30 21:20:34.632245 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:34.632245 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4943da-3bdc-4e2c-900a-5c286a52bd09" containerName="istio-proxy" Oct 30 21:20:34.632432 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:34.632319 4926 cpu_manager.go:414] "containerMap: removing container" podUID="8cc2e803-8410-4ef4-83d7-035cc04f9594" containerName="revoke-keys" Oct 30 21:20:34.632432 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:34.632328 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cc2e803-8410-4ef4-83d7-035cc04f9594" containerName="revoke-keys" Oct 30 21:20:34.632533 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:34.632488 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4943da-3bdc-4e2c-900a-5c286a52bd09" containerName="istio-proxy" Oct 30 21:20:34.632533 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:34.632512 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cc2e803-8410-4ef4-83d7-035cc04f9594" containerName="revoke-keys" Oct 30 21:20:34.636423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:34.636404 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfae3730-snd5s" Oct 30 21:20:34.652114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:34.652095 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfae3730-snd5s"] Oct 30 21:20:34.657480 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:34.657454 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:20:34.657627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:34.657529 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:20:34.708683 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:34.708651 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfae3730-snd5s\" (UID: \"d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfae3730-snd5s" Oct 30 21:20:34.708880 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:34.708783 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfae3730-snd5s\" (UID: \"d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfae3730-snd5s" Oct 30 21:20:34.708880 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:34.708841 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntg9f\" (UniqueName: \"kubernetes.io/projected/d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4-kube-api-access-ntg9f\") pod \"maas-api-revoke-sub-dfae3730-snd5s\" (UID: \"d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfae3730-snd5s" Oct 30 21:20:34.810260 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:34.810229 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfae3730-snd5s\" (UID: \"d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfae3730-snd5s" Oct 30 21:20:34.810417 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:34.810327 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfae3730-snd5s\" (UID: \"d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfae3730-snd5s" Oct 30 21:20:34.810417 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:34.810344 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntg9f\" (UniqueName: \"kubernetes.io/projected/d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4-kube-api-access-ntg9f\") pod \"maas-api-revoke-sub-dfae3730-snd5s\" (UID: \"d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfae3730-snd5s" Oct 30 21:20:34.810996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:34.810965 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dfae3730-snd5s\" (UID: \"d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfae3730-snd5s" Oct 30 21:20:34.812622 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:34.812601 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dfae3730-snd5s\" (UID: \"d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfae3730-snd5s" Oct 30 21:20:34.829509 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:34.829479 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntg9f\" (UniqueName: \"kubernetes.io/projected/d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4-kube-api-access-ntg9f\") pod \"maas-api-revoke-sub-dfae3730-snd5s\" (UID: \"d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfae3730-snd5s" Oct 30 21:20:34.950962 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:34.950933 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfae3730-snd5s" Oct 30 21:20:35.260006 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:20:35.259976 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd670edaa_7a1b_4f0c_83aa_0e6c2ae7daf4.slice/crio-dfebf4ffef9a3fe0d4ce5830f5eb5cf144a912217e520c32e77e753774bd79d5 WatchSource:0}: Error finding container dfebf4ffef9a3fe0d4ce5830f5eb5cf144a912217e520c32e77e753774bd79d5: Status 404 returned error can't find the container with id dfebf4ffef9a3fe0d4ce5830f5eb5cf144a912217e520c32e77e753774bd79d5 Oct 30 21:20:35.261435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:35.261412 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfae3730-snd5s"] Oct 30 21:20:36.191302 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:36.191271 4926 generic.go:356] "Generic (PLEG): container finished" podID="d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4" containerID="e60f88117cd80fc420b1915088d493c2182fc92b9f0d7a975c550ef7ccd953f3" exitCode=0 Oct 30 21:20:36.191699 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:36.191377 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfae3730-snd5s" event={"ID":"d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4","Type":"ContainerDied","Data":"e60f88117cd80fc420b1915088d493c2182fc92b9f0d7a975c550ef7ccd953f3"} Oct 30 21:20:36.191699 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:36.191415 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfae3730-snd5s" event={"ID":"d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4","Type":"ContainerStarted","Data":"dfebf4ffef9a3fe0d4ce5830f5eb5cf144a912217e520c32e77e753774bd79d5"} Oct 30 21:20:37.488481 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:20:37.488445 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:20:37.497519 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:37.497458 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfae3730-snd5s" Oct 30 21:20:37.564535 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:37.561969 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4-kube-api-access-ntg9f\" (UniqueName: \"kubernetes.io/projected/d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4-kube-api-access-ntg9f\") pod \"d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4\" (UID: \"d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4\") " Oct 30 21:20:37.564535 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:37.562190 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4-maas-api-openshift-ca\") pod \"d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4\" (UID: \"d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4\") " Oct 30 21:20:37.564535 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:37.562348 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4-maas-api-xks-ca\") pod \"d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4\" (UID: \"d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4\") " Oct 30 21:20:37.567092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:37.565957 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4-maas-api-openshift-ca" pod "d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4" (UID: "d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:20:37.567092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:37.566247 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4-maas-api-xks-ca" pod "d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4" (UID: "d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:20:37.576421 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:37.573329 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4-kube-api-access-ntg9f" pod "d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4" (UID: "d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4"). InnerVolumeSpecName "kube-api-access-ntg9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:20:37.665841 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:37.665800 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:20:37.665841 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:37.665842 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntg9f\" (UniqueName: \"kubernetes.io/projected/d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4-kube-api-access-ntg9f\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:20:37.666185 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:37.665856 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:20:38.251695 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:38.251669 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfae3730-snd5s" Oct 30 21:20:38.251909 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:38.251699 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dfae3730-snd5s" event={"ID":"d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4","Type":"ContainerDied","Data":"dfebf4ffef9a3fe0d4ce5830f5eb5cf144a912217e520c32e77e753774bd79d5"} Oct 30 21:20:38.251909 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:38.251749 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfebf4ffef9a3fe0d4ce5830f5eb5cf144a912217e520c32e77e753774bd79d5" Oct 30 21:20:43.040150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:43.040112 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f6fad1-n7pln"] Oct 30 21:20:43.048961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:43.048929 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89f6fad1-n7pln"] Oct 30 21:20:45.019225 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:45.019192 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1708326-0038-4cdb-8f8a-5e2577634a66" path="/var/lib/kubelet/pods/c1708326-0038-4cdb-8f8a-5e2577634a66/volumes" Oct 30 21:20:47.296210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:47.296169 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:20:54.135391 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:54.135356 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08e40599-d69vc"] Oct 30 21:20:54.141626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:54.141599 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-08e40599-d69vc"] Oct 30 21:20:55.017881 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:55.017846 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f43dcb80-8b09-473c-b1a5-987225e5fa34" path="/var/lib/kubelet/pods/f43dcb80-8b09-473c-b1a5-987225e5fa34/volumes" Oct 30 21:20:55.071053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:55.066065 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e38050-p9jsh"] Oct 30 21:20:55.088869 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:55.088834 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17e38050-p9jsh"] Oct 30 21:20:57.024106 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:57.024066 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3f8b2fc-80e2-43d2-9509-8927c824e1d7" path="/var/lib/kubelet/pods/d3f8b2fc-80e2-43d2-9509-8927c824e1d7/volumes" Oct 30 21:20:59.030335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.030300 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw"] Oct 30 21:20:59.040742 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.040713 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ee3d8a-ch8mw"] Oct 30 21:20:59.396100 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.395661 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp"] Oct 30 21:20:59.398379 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.396693 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4" containerName="revoke-keys" Oct 30 21:20:59.398379 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.396717 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4" containerName="revoke-keys" Oct 30 21:20:59.398379 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.396753 4926 cpu_manager.go:414] "containerMap: removing container" podUID="3f4943da-3bdc-4e2c-900a-5c286a52bd09" containerName="istio-proxy" Oct 30 21:20:59.398379 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.396760 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4943da-3bdc-4e2c-900a-5c286a52bd09" containerName="istio-proxy" Oct 30 21:20:59.398379 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.396933 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4" containerName="revoke-keys" Oct 30 21:20:59.398379 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.396952 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4943da-3bdc-4e2c-900a-5c286a52bd09" containerName="istio-proxy" Oct 30 21:20:59.403093 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.403069 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.427766 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.427736 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp"] Oct 30 21:20:59.456762 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.456733 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istiod-ca-cert\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.456974 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.456950 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-workload-socket\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.457123 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.457065 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-workload-certs\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.457200 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.457182 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-token\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.457389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.457367 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zblm9\" (UniqueName: \"kubernetes.io/projected/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-kube-api-access-zblm9\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.457525 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.457507 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-credential-socket\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.457594 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.457542 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-data\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.457654 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.457625 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-envoy\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.457710 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.457690 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-podinfo\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.515442 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.515404 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-70d3a9-openshift-default-dockercfg-4h58g\"" Oct 30 21:20:59.562003 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.561964 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-token\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.562537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.562505 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zblm9\" (UniqueName: \"kubernetes.io/projected/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-kube-api-access-zblm9\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.562686 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.562607 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-credential-socket\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.562686 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.562638 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-data\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.562686 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.562675 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-envoy\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.562843 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.562711 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-podinfo\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.563404 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.563373 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-credential-socket\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.563539 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.563436 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istiod-ca-cert\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.563539 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.563480 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-workload-socket\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.563539 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.563517 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-workload-certs\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.563922 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.563898 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-workload-certs\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.565175 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.565153 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istiod-ca-cert\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.565486 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.565464 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-workload-socket\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.565776 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.565756 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-data\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.572096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.569233 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-envoy\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.572703 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.572682 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-podinfo\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.592991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.587732 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-token\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.616051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.610890 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zblm9\" (UniqueName: \"kubernetes.io/projected/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-kube-api-access-zblm9\") pod \"e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:20:59.764107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:20:59.760408 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:21:00.224533 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:00.224494 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:21:00.224969 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:00.224580 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:21:00.224969 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:00.224633 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:21:00.226109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:00.226087 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp"] Oct 30 21:21:00.516146 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:00.515150 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" event={"ID":"d3d50005-aee2-4d9f-b966-7f8a41df5a5a","Type":"ContainerStarted","Data":"0df3de27d0427d80095de282a66195297d01ea84ad9d5dd34536958738e74a1d"} Oct 30 21:21:00.516146 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:00.515218 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" event={"ID":"d3d50005-aee2-4d9f-b966-7f8a41df5a5a","Type":"ContainerStarted","Data":"b6a86c3d3ccf58e623d2536bdd24ce07cd9bbb31ff2ac6c3def9e0a94107fa50"} Oct 30 21:21:00.566255 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:00.566186 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" podStartSLOduration=1.5661659110000001 podStartE2EDuration="1.566165911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:00.55890632 +0000 UTC m=+1247.679980079" watchObservedRunningTime="2026-10-30 21:21:00.566165911 +0000 UTC m=+1247.687239659" Oct 30 21:21:00.761013 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:00.760976 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:21:01.028357 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:01.028316 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f31a01e-712f-4e08-b7ef-f1fe343d368e" path="/var/lib/kubelet/pods/7f31a01e-712f-4e08-b7ef-f1fe343d368e/volumes" Oct 30 21:21:01.767935 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:01.767898 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:21:02.534392 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:02.534359 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:21:02.535507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:02.535484 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:21:07.668056 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:21:07.666775 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:21:09.934053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:09.930657 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx"] Oct 30 21:21:09.992962 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:09.992486 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx"] Oct 30 21:21:10.000087 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:09.998185 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.022324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.020413 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-59fb17-openshift-default-dockercfg-v8czt\"" Oct 30 21:21:10.034971 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.032647 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istiod-ca-cert\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.034971 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.032714 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-envoy\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.034971 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.032745 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-token\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.034971 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.032790 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-workload-certs\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.034971 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.032894 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-podinfo\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.034971 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.032927 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-workload-socket\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.034971 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.032999 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-credential-socket\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.034971 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.033057 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-data\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.034971 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.033094 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bnl5\" (UniqueName: \"kubernetes.io/projected/4aca75f2-2a47-484f-b19b-41c6d228cb2c-kube-api-access-9bnl5\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.134567 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.134516 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-workload-socket\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.134761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.134612 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-credential-socket\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.134761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.134654 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-data\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.134887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.134794 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bnl5\" (UniqueName: \"kubernetes.io/projected/4aca75f2-2a47-484f-b19b-41c6d228cb2c-kube-api-access-9bnl5\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.134945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.134930 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istiod-ca-cert\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.135012 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.134994 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-envoy\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.135116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.135045 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-token\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.135116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.135063 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-workload-socket\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.135116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.135111 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-workload-certs\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.135303 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.135115 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-data\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.135303 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.135132 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-credential-socket\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.135303 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.135271 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-podinfo\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.135458 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.135367 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-workload-certs\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.135747 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.135723 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istiod-ca-cert\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.137981 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.137953 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-envoy\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.138343 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.138317 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-podinfo\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.160799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.160769 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bnl5\" (UniqueName: \"kubernetes.io/projected/4aca75f2-2a47-484f-b19b-41c6d228cb2c-kube-api-access-9bnl5\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.161071 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.161047 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-token\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.340712 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.340621 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:10.706685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.706654 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx"] Oct 30 21:21:10.706835 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:21:10.706812 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aca75f2_2a47_484f_b19b_41c6d228cb2c.slice/crio-dc12adc1a0c57ce5ee41ce0f2796973872c00d233ad27124e04d9df92ded7cc6 WatchSource:0}: Error finding container dc12adc1a0c57ce5ee41ce0f2796973872c00d233ad27124e04d9df92ded7cc6: Status 404 returned error can't find the container with id dc12adc1a0c57ce5ee41ce0f2796973872c00d233ad27124e04d9df92ded7cc6 Oct 30 21:21:10.709437 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.709408 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:21:10.709532 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.709466 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:21:10.709532 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:10.709496 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:21:11.044247 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.044154 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775"] Oct 30 21:21:11.049986 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.049957 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.077096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.073842 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0e7510-openshift-default-dockercfg-5kn2j\"" Oct 30 21:21:11.077096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.075808 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775"] Oct 30 21:21:11.159611 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.159570 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-workload-certs\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.159792 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.159644 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-credential-socket\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.159926 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.159822 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istiod-ca-cert\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.159926 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.159855 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrqcs\" (UniqueName: \"kubernetes.io/projected/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-kube-api-access-lrqcs\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.160103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.159929 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-data\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.160103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.159972 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-token\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.160103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.160058 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-envoy\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.160326 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.160166 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-workload-socket\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.160326 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.160197 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-podinfo\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.264201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.262112 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-data\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.264201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.262181 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-token\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.264201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.262258 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-envoy\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.264201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.262295 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-workload-socket\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.264201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.262321 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-podinfo\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.264201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.262437 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-workload-certs\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.264201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.262503 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-credential-socket\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.264201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.262671 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istiod-ca-cert\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.264201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.262707 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrqcs\" (UniqueName: \"kubernetes.io/projected/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-kube-api-access-lrqcs\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.264201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.263539 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-data\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.268174 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.267517 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-envoy\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.268174 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.267789 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-workload-socket\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.277067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.271094 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istiod-ca-cert\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.277067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.271355 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-workload-certs\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.277067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.275824 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-podinfo\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.277067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.276313 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-credential-socket\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.296106 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.293128 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrqcs\" (UniqueName: \"kubernetes.io/projected/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-kube-api-access-lrqcs\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.301051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.299562 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-token\") pod \"e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.392057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.389301 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:11.716861 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.716824 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" event={"ID":"4aca75f2-2a47-484f-b19b-41c6d228cb2c","Type":"ContainerStarted","Data":"dc884dc4bf22f01ff10c78a8c7cf11fde75968bea27bd1aec3e593014052384e"} Oct 30 21:21:11.717074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.716882 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" event={"ID":"4aca75f2-2a47-484f-b19b-41c6d228cb2c","Type":"ContainerStarted","Data":"dc12adc1a0c57ce5ee41ce0f2796973872c00d233ad27124e04d9df92ded7cc6"} Oct 30 21:21:11.762437 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.762370 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" podStartSLOduration=2.762349726 podStartE2EDuration="2.762349726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:11.756479843 +0000 UTC m=+1258.877553590" watchObservedRunningTime="2026-10-30 21:21:11.762349726 +0000 UTC m=+1258.883423477" Oct 30 21:21:11.848280 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.848248 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775"] Oct 30 21:21:11.849151 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:21:11.849123 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c6b634e_2a0b_4c0a_aa07_7901e40d77db.slice/crio-d32e9aa19b0f8423e118025adaac232238433501faaa5a771bdc71758268a4da WatchSource:0}: Error finding container d32e9aa19b0f8423e118025adaac232238433501faaa5a771bdc71758268a4da: Status 404 returned error can't find the container with id d32e9aa19b0f8423e118025adaac232238433501faaa5a771bdc71758268a4da Oct 30 21:21:11.852532 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.852493 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:21:11.852624 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.852579 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:21:11.852665 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:11.852630 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:21:12.341603 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:12.341570 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:12.348162 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:12.348127 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:12.729827 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:12.729782 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" event={"ID":"8c6b634e-2a0b-4c0a-aa07-7901e40d77db","Type":"ContainerStarted","Data":"140ab3d0fb774e808bd48d46b3bc9e9165024f96dc38bd16ca136012d9f4c481"} Oct 30 21:21:12.730080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:12.729851 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" event={"ID":"8c6b634e-2a0b-4c0a-aa07-7901e40d77db","Type":"ContainerStarted","Data":"d32e9aa19b0f8423e118025adaac232238433501faaa5a771bdc71758268a4da"} Oct 30 21:21:12.730311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:12.730286 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:12.731736 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:12.731711 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:21:12.772788 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:12.772729 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" podStartSLOduration=1.7727149089999998 podStartE2EDuration="1.772714909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:12.767890861 +0000 UTC m=+1259.888964608" watchObservedRunningTime="2026-10-30 21:21:12.772714909 +0000 UTC m=+1259.893788654" Oct 30 21:21:13.390070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:13.389996 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:13.397757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:13.397464 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:13.536767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:13.536312 4926 scope.go:122] "RemoveContainer" containerID="31c95b5888ed910a6a42deecd69621ceb7f02166552c97c29b4a7f42f61f8d03" Oct 30 21:21:13.553498 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:13.553476 4926 scope.go:122] "RemoveContainer" containerID="62f49b4607fb88b1375586d389d58e99121814e8d1e8e85799984cb0ccfffff5" Oct 30 21:21:13.566678 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:13.566651 4926 scope.go:122] "RemoveContainer" containerID="07c253b136a6dce0b2deb21e241c6e343510bfc35ab4629a440377d6210cb428" Oct 30 21:21:13.743220 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:13.743191 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:13.744302 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:13.744279 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:21:15.403566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.403519 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4"] Oct 30 21:21:15.413417 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.413384 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.433856 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.433820 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4"] Oct 30 21:21:15.456345 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.434738 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-a4782a-openshift-default-dockercfg-5x6tr\"" Oct 30 21:21:15.482208 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.482115 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-workload-socket\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.482208 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.482181 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-envoy\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.482437 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.482273 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-workload-certs\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.482437 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.482375 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-podinfo\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.482437 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.482422 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-data\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.482596 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.482523 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twpsd\" (UniqueName: \"kubernetes.io/projected/278c1234-a1ec-4205-b4be-b0d567c6cd10-kube-api-access-twpsd\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.482596 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.482591 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-credential-socket\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.482734 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.482714 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-token\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.482790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.482762 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278c1234-a1ec-4205-b4be-b0d567c6cd10-istiod-ca-cert\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.595055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.590157 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-token\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.595055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.590219 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278c1234-a1ec-4205-b4be-b0d567c6cd10-istiod-ca-cert\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.595055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.590342 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-workload-socket\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.595055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.590384 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-envoy\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.595055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.590408 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-workload-certs\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.595055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.590471 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-podinfo\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.595055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.590515 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-data\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.595055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.590619 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twpsd\" (UniqueName: \"kubernetes.io/projected/278c1234-a1ec-4205-b4be-b0d567c6cd10-kube-api-access-twpsd\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.595055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.590688 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-credential-socket\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.595055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.591292 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-credential-socket\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.595055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.592286 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278c1234-a1ec-4205-b4be-b0d567c6cd10-istiod-ca-cert\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.595055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.592543 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-workload-socket\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.595055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.593521 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-workload-certs\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.595055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.593789 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-data\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.631055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.626839 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-envoy\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.651057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.648529 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-podinfo\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.661109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.660661 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-token\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.666799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.661306 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twpsd\" (UniqueName: \"kubernetes.io/projected/278c1234-a1ec-4205-b4be-b0d567c6cd10-kube-api-access-twpsd\") pod \"e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:15.772055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:15.771511 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:16.204291 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:16.204246 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4"] Oct 30 21:21:16.206259 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:21:16.206226 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278c1234_a1ec_4205_b4be_b0d567c6cd10.slice/crio-6e9062339543ee54de343f2c0d232bae34464cd927922512ee2af84c302a4384 WatchSource:0}: Error finding container 6e9062339543ee54de343f2c0d232bae34464cd927922512ee2af84c302a4384: Status 404 returned error can't find the container with id 6e9062339543ee54de343f2c0d232bae34464cd927922512ee2af84c302a4384 Oct 30 21:21:16.209692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:16.209655 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:21:16.209801 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:16.209733 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:21:16.209801 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:16.209765 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:21:16.822060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:16.819954 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" event={"ID":"278c1234-a1ec-4205-b4be-b0d567c6cd10","Type":"ContainerStarted","Data":"7bd4150b5406693c32b80b057d6d0f3c529988d734aa4618dcbbcb5570d29dad"} Oct 30 21:21:16.822060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:16.820015 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" event={"ID":"278c1234-a1ec-4205-b4be-b0d567c6cd10","Type":"ContainerStarted","Data":"6e9062339543ee54de343f2c0d232bae34464cd927922512ee2af84c302a4384"} Oct 30 21:21:17.772603 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:17.772561 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:17.779514 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:17.779486 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:17.830568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:17.830525 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:17.831573 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:17.831549 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:21:17.839640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:17.839594 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" podStartSLOduration=2.83958079 podStartE2EDuration="2.83958079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:16.883086808 +0000 UTC m=+1264.004160555" watchObservedRunningTime="2026-10-30 21:21:17.83958079 +0000 UTC m=+1264.960654601" Oct 30 21:21:20.032765 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:20.032733 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq"] Oct 30 21:21:20.043136 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:20.043108 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c14822df-7m7pq"] Oct 30 21:21:21.020315 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:21.020277 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cbd47fc-81f6-46b9-8916-76b6e5cf7002" path="/var/lib/kubelet/pods/6cbd47fc-81f6-46b9-8916-76b6e5cf7002/volumes" Oct 30 21:21:28.172264 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:28.172229 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:21:35.036366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:35.036334 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e642b143-pt4f5"] Oct 30 21:21:35.046807 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:35.046775 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e642b143-pt4f5"] Oct 30 21:21:37.018226 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:37.018191 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f395eae6-6fea-4108-9022-e52ed87ca24d" path="/var/lib/kubelet/pods/f395eae6-6fea-4108-9022-e52ed87ca24d/volumes" Oct 30 21:21:37.485942 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:21:37.485901 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:21:45.290379 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.290347 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz"] Oct 30 21:21:45.296017 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.295998 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.333685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.333653 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz"] Oct 30 21:21:45.430725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.430686 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hd6sz\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.430884 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.430859 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d02b4c5e-92d2-412b-8073-da97d6fb4a60-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hd6sz\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.431097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.431075 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d02b4c5e-92d2-412b-8073-da97d6fb4a60-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hd6sz\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.431202 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.431189 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hd6sz\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.431315 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.431277 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hd6sz\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.431400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.431380 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hd6sz\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.431441 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.431422 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkm77\" (UniqueName: \"kubernetes.io/projected/d02b4c5e-92d2-412b-8073-da97d6fb4a60-kube-api-access-rkm77\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hd6sz\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.532513 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.532453 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d02b4c5e-92d2-412b-8073-da97d6fb4a60-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hd6sz\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.532704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.532637 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d02b4c5e-92d2-412b-8073-da97d6fb4a60-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hd6sz\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.532777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.532718 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hd6sz\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.532777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.532744 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hd6sz\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.532889 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.532805 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hd6sz\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.532889 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.532852 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkm77\" (UniqueName: \"kubernetes.io/projected/d02b4c5e-92d2-412b-8073-da97d6fb4a60-kube-api-access-rkm77\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hd6sz\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.533116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.533089 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hd6sz\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.533703 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.533673 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hd6sz\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.535017 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.534996 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d02b4c5e-92d2-412b-8073-da97d6fb4a60-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hd6sz\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.535231 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.535200 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hd6sz\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.535513 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.535495 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hd6sz\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.535560 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.535522 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d02b4c5e-92d2-412b-8073-da97d6fb4a60-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hd6sz\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.552372 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.552296 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkm77\" (UniqueName: \"kubernetes.io/projected/d02b4c5e-92d2-412b-8073-da97d6fb4a60-kube-api-access-rkm77\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hd6sz\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.552496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.552468 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-hd6sz\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.611521 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.611496 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:45.941203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.941167 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz"] Oct 30 21:21:45.952769 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:21:45.952742 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd02b4c5e_92d2_412b_8073_da97d6fb4a60.slice/crio-5300bd5526987c91ec915b73434bee6791fcba6130f592cd7d48b844ccbe3251 WatchSource:0}: Error finding container 5300bd5526987c91ec915b73434bee6791fcba6130f592cd7d48b844ccbe3251: Status 404 returned error can't find the container with id 5300bd5526987c91ec915b73434bee6791fcba6130f592cd7d48b844ccbe3251 Oct 30 21:21:45.955323 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.955288 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:21:45.955415 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:45.955351 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:21:46.165253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:46.165212 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" event={"ID":"d02b4c5e-92d2-412b-8073-da97d6fb4a60","Type":"ContainerStarted","Data":"58883ae13a4f930fb19bd7f7be4cd5e7cb44360b42a925452213072789005a9f"} Oct 30 21:21:46.165431 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:46.165282 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" event={"ID":"d02b4c5e-92d2-412b-8073-da97d6fb4a60","Type":"ContainerStarted","Data":"5300bd5526987c91ec915b73434bee6791fcba6130f592cd7d48b844ccbe3251"} Oct 30 21:21:46.165431 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:46.165323 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:46.223637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:46.223537 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" podStartSLOduration=1.223520844 podStartE2EDuration="1.223520844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:21:46.210700044 +0000 UTC m=+1293.331773791" watchObservedRunningTime="2026-10-30 21:21:46.223520844 +0000 UTC m=+1293.344594591" Oct 30 21:21:47.182289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:47.182254 4926 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-hd6sz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:21:47.182680 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:47.182315 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" podUID="d02b4c5e-92d2-412b-8073-da97d6fb4a60" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:21:50.180549 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:50.180516 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:21:53.029546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:53.029512 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn"] Oct 30 21:21:53.039623 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:53.039596 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c49ffb9b-j9dpn"] Oct 30 21:21:55.022166 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:21:55.022050 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8491486d-cb78-40a4-ab36-7ea5ece8a5a0" path="/var/lib/kubelet/pods/8491486d-cb78-40a4-ab36-7ea5ece8a5a0/volumes" Oct 30 21:22:05.022490 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:05.022428 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:22:05.162264 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:05.162153 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 21:22:05.174673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:05.174632 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:05.244271 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:05.244196 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:05.512493 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:05.512213 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:05.552718 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:05.552117 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:05.580736 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:05.580691 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:05.590521 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:05.581431 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:05.650594 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:05.650172 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:05.841222 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:05.841121 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:05.900120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:05.896780 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:06.111116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:06.042127 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:06.111116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:06.042714 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:22:06.445754 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:06.445707 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:06.447809 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:06.447780 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:22:06.551787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:06.551742 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded" Oct 30 21:22:06.573218 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:06.573172 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded" Oct 30 21:22:06.694055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:06.612312 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 21:22:06.720154 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:06.715594 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:22:06.748629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:06.741841 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 21:22:06.786492 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:06.771384 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:22:06.878058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:06.865100 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:22:07.069112 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:07.051126 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:22:07.069112 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:07.059901 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 21:22:07.069112 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:07.062726 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:22:07.069112 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:07.062785 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:22:07.090976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:07.090931 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:22:07.106056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:07.092421 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:07.114497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:07.110535 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8k468" podUID="97d86e50-7f90-4650-bef4-2a4cd91c1d61" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.37:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:07.173575 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:07.168762 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:07.412136 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:07.409206 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:22:07.412136 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:07.409273 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:22:07.446316 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:07.446277 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:07.735114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:07.697196 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:07.813379 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:07.813329 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:08.051858 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:08.051763 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:22:08.069405 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:08.069357 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:08.078015 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:08.077990 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:22:08.112404 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:08.112364 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:08.112899 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:08.112863 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:08.114371 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:08.114333 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:08.114819 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:08.114795 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:22:08.116818 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:08.116786 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:08.173670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:08.173628 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 21:22:08.174698 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:08.174672 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:22:08.252460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:08.249892 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:08.342781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:08.324360 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:08.362425 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:08.348109 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:08.430067 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:22:08.429994 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:22:08.455058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:08.453297 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:08.468055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:08.466648 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 21:22:08.733863 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:08.733820 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:08.790059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:08.773885 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:09.046560 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:09.032807 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:09.176078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:09.161157 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:09.177780 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:09.177738 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:09.202187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:09.202131 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:22:09.221786 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:09.221746 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:22:09.284055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:09.222457 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 21:22:09.308115 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:09.269443 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 21:22:09.308115 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:09.296188 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:09.315057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:09.310717 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:22:09.315057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:09.310786 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:22:09.315057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:09.310820 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:22:09.373057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:09.348161 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:09.378637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:09.376122 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:22:09.378637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:09.352745 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:09.378857 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:09.378837 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:22:09.461058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:09.461001 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:09.501760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:09.477119 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:09.529307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:09.529273 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:22:09.735365 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:09.735320 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 21:22:09.909142 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:09.785193 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:09.910171 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:09.910141 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:22:10.063898 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:10.063789 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:10.178058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:10.156411 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:22:10.178058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:10.161976 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:10.281091 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:10.225624 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:10.281091 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:10.226433 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:10.281091 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:10.226668 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 21:22:10.336110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:10.321563 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:10.397673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:10.397532 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 21:22:10.401399 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:10.401361 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:22:10.498080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:10.471248 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:10.572193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:10.572132 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:10.633114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:10.632165 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 21:22:10.681996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:10.681115 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 21:22:10.728868 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:10.728833 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 21:22:10.923112 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:10.914238 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:22:11.196115 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:11.195439 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:11.223174 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:11.197013 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:11.248059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:11.243697 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:11.248059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:11.243577 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:11.248059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:11.245879 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:22:11.350099 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:11.350063 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:22:11.354180 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:11.354148 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:22:11.354409 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:11.354395 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:22:11.411055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:11.400349 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:11.411055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:11.403908 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:11.474895 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:11.474798 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:11.584657 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:11.584605 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:11.678058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:11.664178 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:11.785385 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:11.778740 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 21:22:11.882211 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:11.870677 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:11.944059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:11.917106 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:12.124126 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:12.121331 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:22:12.187005 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:12.177937 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:12.199162 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:12.199114 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 21:22:12.207152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:12.207109 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:12.245190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:12.245139 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:12.246399 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:12.246364 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:22:12.298221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:12.287834 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:12.298221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:12.288225 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 21:22:12.298221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:12.288783 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded" Oct 30 21:22:12.335423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:12.333328 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:22:12.415661 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:12.415474 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:12.416275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:12.416245 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 21:22:12.417502 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:12.417483 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:22:12.523068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:12.493572 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:12.548344 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:12.548289 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:12.591061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:12.560264 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": context deadline exceeded" Oct 30 21:22:12.591473 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:12.591360 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:12.591473 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:12.591428 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:12.796434 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:12.796296 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:12.807675 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:12.807638 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:12.813475 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:12.813448 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:22:13.240240 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:13.240195 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:13.251840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:13.245985 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:13.298052 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:13.293397 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 21:22:13.298052 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:13.293869 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:22:13.300627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:13.300003 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:13.300627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:13.300397 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:22:13.321943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:13.321913 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:22:13.426055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:13.401880 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:22:13.426055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:13.422139 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:13.437074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:13.429352 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:22:13.521109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:13.517196 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:13.601478 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:13.601435 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:13.666925 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:13.659465 4926 scope.go:122] "RemoveContainer" containerID="70808953ead605130868f860042b3b777f8562e1f13adcd0c56b3800e73bc923" Oct 30 21:22:13.819106 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:13.811435 4926 scope.go:122] "RemoveContainer" containerID="bd144cb66f81eb4af57d8602a29f17da7dd86d4dcba768d70ac6aa68b47dd429" Oct 30 21:22:13.819856 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:13.819813 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:13.852923 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:13.852864 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 21:22:13.853963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:13.853928 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:22:13.869342 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:13.869289 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:14.013319 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:14.013265 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:14.269079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:14.262404 4926 scope.go:122] "RemoveContainer" containerID="6c2e2fd2b93646b708df51b920e5bbfd662548cc01c72b4adf667806aa726aa7" Oct 30 21:22:14.316702 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:14.309566 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:14.325376 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:14.325326 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:14.341797 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:14.341750 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 21:22:14.394197 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:14.394156 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:14.434906 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:14.434859 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:14.468351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:14.468323 4926 scope.go:122] "RemoveContainer" containerID="1812dcf0995ac4f1ae2070dc062c48f1a02ca141afbc2f350b7e5262f0dd0e33" Oct 30 21:22:14.549171 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:14.539431 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 21:22:14.549171 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:14.539816 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": context deadline exceeded" Oct 30 21:22:14.549171 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:14.548742 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:14.559468 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:14.559084 4926 scope.go:122] "RemoveContainer" containerID="4b171cc43f13ce72af3b4daec1d55ffc03b96ec9951b79a326f2e00434deb2a0" Oct 30 21:22:14.578310 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:14.577982 4926 scope.go:122] "RemoveContainer" containerID="c68dec1b952c277b71954f641042d0adda85f5f26305b1d0ad866e4db66b6121" Oct 30 21:22:14.650084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:14.643711 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:22:14.722420 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:14.722395 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:22:14.799194 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:14.796449 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:14.832177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:14.832001 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:14.879212 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:14.879166 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 21:22:15.082070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.067128 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:15.082070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.067495 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 21:22:15.317112 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.316321 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:15.317608 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.317305 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 21:22:15.422385 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.378504 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:15.422946 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.414776 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:15.512098 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.510116 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:15.550125 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.548904 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:15.562231 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.561719 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:15.636785 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.636619 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps"] Oct 30 21:22:15.709533 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.709501 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:22:15.709689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.709560 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:22:15.709689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.709593 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:22:15.709689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.709624 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:22:15.710459 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.710431 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps" Oct 30 21:22:15.714673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.714621 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:22:15.714859 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.714844 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:22:15.740092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.738934 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps"] Oct 30 21:22:15.740092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.739205 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:22:15.740092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.739374 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:22:15.740448 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.740432 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:22:15.799238 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.799213 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:22:15.883960 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.881821 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:15.910405 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.904089 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4951ebd2-7b73-442d-a1d5-1e517970c885-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a04cffd9-62zps\" (UID: \"4951ebd2-7b73-442d-a1d5-1e517970c885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps" Oct 30 21:22:15.916411 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.915887 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:22:15.919707 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.917651 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4951ebd2-7b73-442d-a1d5-1e517970c885-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a04cffd9-62zps\" (UID: \"4951ebd2-7b73-442d-a1d5-1e517970c885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps" Oct 30 21:22:15.919707 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:15.917822 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlzrs\" (UniqueName: \"kubernetes.io/projected/4951ebd2-7b73-442d-a1d5-1e517970c885-kube-api-access-dlzrs\") pod \"maas-api-revoke-sub-a04cffd9-62zps\" (UID: \"4951ebd2-7b73-442d-a1d5-1e517970c885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps" Oct 30 21:22:16.021345 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:16.021303 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlzrs\" (UniqueName: \"kubernetes.io/projected/4951ebd2-7b73-442d-a1d5-1e517970c885-kube-api-access-dlzrs\") pod \"maas-api-revoke-sub-a04cffd9-62zps\" (UID: \"4951ebd2-7b73-442d-a1d5-1e517970c885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps" Oct 30 21:22:16.021587 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:16.021564 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4951ebd2-7b73-442d-a1d5-1e517970c885-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a04cffd9-62zps\" (UID: \"4951ebd2-7b73-442d-a1d5-1e517970c885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps" Oct 30 21:22:16.021668 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:16.021653 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4951ebd2-7b73-442d-a1d5-1e517970c885-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a04cffd9-62zps\" (UID: \"4951ebd2-7b73-442d-a1d5-1e517970c885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps" Oct 30 21:22:16.022562 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:16.022530 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4951ebd2-7b73-442d-a1d5-1e517970c885-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a04cffd9-62zps\" (UID: \"4951ebd2-7b73-442d-a1d5-1e517970c885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps" Oct 30 21:22:16.030856 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:16.030751 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4951ebd2-7b73-442d-a1d5-1e517970c885-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a04cffd9-62zps\" (UID: \"4951ebd2-7b73-442d-a1d5-1e517970c885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps" Oct 30 21:22:16.043061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:16.042987 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlzrs\" (UniqueName: \"kubernetes.io/projected/4951ebd2-7b73-442d-a1d5-1e517970c885-kube-api-access-dlzrs\") pod \"maas-api-revoke-sub-a04cffd9-62zps\" (UID: \"4951ebd2-7b73-442d-a1d5-1e517970c885\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps" Oct 30 21:22:16.340634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:16.340596 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps" Oct 30 21:22:17.384493 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:17.336454 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps"] Oct 30 21:22:17.730699 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:17.730511 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps" event={"ID":"4951ebd2-7b73-442d-a1d5-1e517970c885","Type":"ContainerStarted","Data":"56771687f8c5b413c079f8e20ffbb28c47fb69744333a00be5b822f757d2f966"} Oct 30 21:22:18.579541 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:18.579167 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:22:18.579541 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:18.579229 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" podUID="d3d50005-aee2-4d9f-b966-7f8a41df5a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:18.826331 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:18.826288 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps" event={"ID":"4951ebd2-7b73-442d-a1d5-1e517970c885","Type":"ContainerStarted","Data":"2cd0fce63212f84feb0ee8aa9685a3eb562d9830d29f2cfd8323f3b7df046ec6"} Oct 30 21:22:18.884147 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:18.875123 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:22:18.884147 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:18.875186 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" podUID="278c1234-a1ec-4205-b4be-b0d567c6cd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:18.897496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:18.886093 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps" podStartSLOduration=3.886071232 podStartE2EDuration="3.886071232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:18.874633466 +0000 UTC m=+1325.995707210" watchObservedRunningTime="2026-10-30 21:22:18.886071232 +0000 UTC m=+1326.007144979" Oct 30 21:22:19.865055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:19.862452 4926 generic.go:356] "Generic (PLEG): container finished" podID="4951ebd2-7b73-442d-a1d5-1e517970c885" containerID="2cd0fce63212f84feb0ee8aa9685a3eb562d9830d29f2cfd8323f3b7df046ec6" exitCode=0 Oct 30 21:22:19.865055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:19.862745 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps" event={"ID":"4951ebd2-7b73-442d-a1d5-1e517970c885","Type":"ContainerDied","Data":"2cd0fce63212f84feb0ee8aa9685a3eb562d9830d29f2cfd8323f3b7df046ec6"} Oct 30 21:22:21.374597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:21.374557 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps" Oct 30 21:22:21.446996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:21.446961 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4951ebd2-7b73-442d-a1d5-1e517970c885-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4951ebd2-7b73-442d-a1d5-1e517970c885-maas-api-xks-ca\") pod \"4951ebd2-7b73-442d-a1d5-1e517970c885\" (UID: \"4951ebd2-7b73-442d-a1d5-1e517970c885\") " Oct 30 21:22:21.447220 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:21.447066 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4951ebd2-7b73-442d-a1d5-1e517970c885-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4951ebd2-7b73-442d-a1d5-1e517970c885-maas-api-openshift-ca\") pod \"4951ebd2-7b73-442d-a1d5-1e517970c885\" (UID: \"4951ebd2-7b73-442d-a1d5-1e517970c885\") " Oct 30 21:22:21.447355 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:21.447326 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4951ebd2-7b73-442d-a1d5-1e517970c885-kube-api-access-dlzrs\" (UniqueName: \"kubernetes.io/projected/4951ebd2-7b73-442d-a1d5-1e517970c885-kube-api-access-dlzrs\") pod \"4951ebd2-7b73-442d-a1d5-1e517970c885\" (UID: \"4951ebd2-7b73-442d-a1d5-1e517970c885\") " Oct 30 21:22:21.447805 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:21.447745 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4951ebd2-7b73-442d-a1d5-1e517970c885-maas-api-openshift-ca" pod "4951ebd2-7b73-442d-a1d5-1e517970c885" (UID: "4951ebd2-7b73-442d-a1d5-1e517970c885"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:22:21.448154 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:21.448131 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4951ebd2-7b73-442d-a1d5-1e517970c885-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:22:21.456007 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:21.455963 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4951ebd2-7b73-442d-a1d5-1e517970c885-maas-api-xks-ca" pod "4951ebd2-7b73-442d-a1d5-1e517970c885" (UID: "4951ebd2-7b73-442d-a1d5-1e517970c885"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:22:21.457596 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:21.457556 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4951ebd2-7b73-442d-a1d5-1e517970c885-kube-api-access-dlzrs" pod "4951ebd2-7b73-442d-a1d5-1e517970c885" (UID: "4951ebd2-7b73-442d-a1d5-1e517970c885"). InnerVolumeSpecName "kube-api-access-dlzrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:22:21.550282 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:21.550000 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4951ebd2-7b73-442d-a1d5-1e517970c885-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:22:21.550282 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:21.550056 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlzrs\" (UniqueName: \"kubernetes.io/projected/4951ebd2-7b73-442d-a1d5-1e517970c885-kube-api-access-dlzrs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:22:21.956094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:21.955562 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps" event={"ID":"4951ebd2-7b73-442d-a1d5-1e517970c885","Type":"ContainerDied","Data":"56771687f8c5b413c079f8e20ffbb28c47fb69744333a00be5b822f757d2f966"} Oct 30 21:22:21.956094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:21.955624 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56771687f8c5b413c079f8e20ffbb28c47fb69744333a00be5b822f757d2f966" Oct 30 21:22:21.956094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:21.955714 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps" Oct 30 21:22:22.720223 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:22.715790 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-s82n5"] Oct 30 21:22:22.739754 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:22.736909 4926 cpu_manager.go:414] "containerMap: removing container" podUID="4951ebd2-7b73-442d-a1d5-1e517970c885" containerName="revoke-keys" Oct 30 21:22:22.739754 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:22.736940 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4951ebd2-7b73-442d-a1d5-1e517970c885" containerName="revoke-keys" Oct 30 21:22:22.739754 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:22.737376 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="4951ebd2-7b73-442d-a1d5-1e517970c885" containerName="revoke-keys" Oct 30 21:22:22.818145 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:22.818108 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-s82n5"] Oct 30 21:22:22.818513 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:22.818490 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-s82n5" Oct 30 21:22:22.849792 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:22.849749 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:22:22.862230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:22.862182 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:22:22.922670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:22.922627 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ebc545e-7736-47fb-af0d-ab6cfd748fad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1da2183-s82n5\" (UID: \"8ebc545e-7736-47fb-af0d-ab6cfd748fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-s82n5" Oct 30 21:22:22.922876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:22.922822 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgmg7\" (UniqueName: \"kubernetes.io/projected/8ebc545e-7736-47fb-af0d-ab6cfd748fad-kube-api-access-qgmg7\") pod \"maas-api-revoke-keys-e2e-mt-c1da2183-s82n5\" (UID: \"8ebc545e-7736-47fb-af0d-ab6cfd748fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-s82n5" Oct 30 21:22:22.922945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:22.922893 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ebc545e-7736-47fb-af0d-ab6cfd748fad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1da2183-s82n5\" (UID: \"8ebc545e-7736-47fb-af0d-ab6cfd748fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-s82n5" Oct 30 21:22:23.028144 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:23.027777 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ebc545e-7736-47fb-af0d-ab6cfd748fad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1da2183-s82n5\" (UID: \"8ebc545e-7736-47fb-af0d-ab6cfd748fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-s82n5" Oct 30 21:22:23.028144 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:23.027912 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgmg7\" (UniqueName: \"kubernetes.io/projected/8ebc545e-7736-47fb-af0d-ab6cfd748fad-kube-api-access-qgmg7\") pod \"maas-api-revoke-keys-e2e-mt-c1da2183-s82n5\" (UID: \"8ebc545e-7736-47fb-af0d-ab6cfd748fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-s82n5" Oct 30 21:22:23.028144 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:23.027992 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ebc545e-7736-47fb-af0d-ab6cfd748fad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1da2183-s82n5\" (UID: \"8ebc545e-7736-47fb-af0d-ab6cfd748fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-s82n5" Oct 30 21:22:23.029337 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:23.029267 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ebc545e-7736-47fb-af0d-ab6cfd748fad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1da2183-s82n5\" (UID: \"8ebc545e-7736-47fb-af0d-ab6cfd748fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-s82n5" Oct 30 21:22:23.039015 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:23.038958 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ebc545e-7736-47fb-af0d-ab6cfd748fad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1da2183-s82n5\" (UID: \"8ebc545e-7736-47fb-af0d-ab6cfd748fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-s82n5" Oct 30 21:22:23.052663 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:23.052600 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgmg7\" (UniqueName: \"kubernetes.io/projected/8ebc545e-7736-47fb-af0d-ab6cfd748fad-kube-api-access-qgmg7\") pod \"maas-api-revoke-keys-e2e-mt-c1da2183-s82n5\" (UID: \"8ebc545e-7736-47fb-af0d-ab6cfd748fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-s82n5" Oct 30 21:22:23.195707 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:23.195663 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-s82n5" Oct 30 21:22:23.647262 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:23.630920 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-s82n5"] Oct 30 21:22:24.022056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:23.989363 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-s82n5" event={"ID":"8ebc545e-7736-47fb-af0d-ab6cfd748fad","Type":"ContainerStarted","Data":"3e6ea3dac26c1fdfc6eb98e061146c40632286501b58983815dea162cf2dd84f"} Oct 30 21:22:25.003079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:25.002171 4926 generic.go:356] "Generic (PLEG): container finished" podID="8ebc545e-7736-47fb-af0d-ab6cfd748fad" containerID="fc2880c7cb5096c5ab5141bd37cbcf6c20fbc417c8b2a44761fb45c11b886d25" exitCode=0 Oct 30 21:22:25.003079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:25.002388 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-s82n5" event={"ID":"8ebc545e-7736-47fb-af0d-ab6cfd748fad","Type":"ContainerDied","Data":"fc2880c7cb5096c5ab5141bd37cbcf6c20fbc417c8b2a44761fb45c11b886d25"} Oct 30 21:22:26.029238 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:26.029200 4926 generic.go:356] "Generic (PLEG): container finished" podID="278c1234-a1ec-4205-b4be-b0d567c6cd10" containerID="7bd4150b5406693c32b80b057d6d0f3c529988d734aa4618dcbbcb5570d29dad" exitCode=0 Oct 30 21:22:26.067630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:26.030183 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" event={"ID":"278c1234-a1ec-4205-b4be-b0d567c6cd10","Type":"ContainerDied","Data":"7bd4150b5406693c32b80b057d6d0f3c529988d734aa4618dcbbcb5570d29dad"} Oct 30 21:22:26.067630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:26.032776 4926 scope.go:122] "RemoveContainer" containerID="7bd4150b5406693c32b80b057d6d0f3c529988d734aa4618dcbbcb5570d29dad" Oct 30 21:22:26.067630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:26.037163 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:22:26.067630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:26.037307 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:22:26.067630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:26.037382 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:22:26.315816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:26.315501 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-s82n5" Oct 30 21:22:26.358717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:26.358682 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ebc545e-7736-47fb-af0d-ab6cfd748fad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ebc545e-7736-47fb-af0d-ab6cfd748fad-maas-api-xks-ca\") pod \"8ebc545e-7736-47fb-af0d-ab6cfd748fad\" (UID: \"8ebc545e-7736-47fb-af0d-ab6cfd748fad\") " Oct 30 21:22:26.358874 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:26.358798 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ebc545e-7736-47fb-af0d-ab6cfd748fad-kube-api-access-qgmg7\" (UniqueName: \"kubernetes.io/projected/8ebc545e-7736-47fb-af0d-ab6cfd748fad-kube-api-access-qgmg7\") pod \"8ebc545e-7736-47fb-af0d-ab6cfd748fad\" (UID: \"8ebc545e-7736-47fb-af0d-ab6cfd748fad\") " Oct 30 21:22:26.358941 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:26.358892 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ebc545e-7736-47fb-af0d-ab6cfd748fad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ebc545e-7736-47fb-af0d-ab6cfd748fad-maas-api-openshift-ca\") pod \"8ebc545e-7736-47fb-af0d-ab6cfd748fad\" (UID: \"8ebc545e-7736-47fb-af0d-ab6cfd748fad\") " Oct 30 21:22:26.359369 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:26.359343 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ebc545e-7736-47fb-af0d-ab6cfd748fad-maas-api-openshift-ca" pod "8ebc545e-7736-47fb-af0d-ab6cfd748fad" (UID: "8ebc545e-7736-47fb-af0d-ab6cfd748fad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:22:26.359838 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:26.359818 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ebc545e-7736-47fb-af0d-ab6cfd748fad-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:22:26.361161 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:26.361135 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ebc545e-7736-47fb-af0d-ab6cfd748fad-kube-api-access-qgmg7" pod "8ebc545e-7736-47fb-af0d-ab6cfd748fad" (UID: "8ebc545e-7736-47fb-af0d-ab6cfd748fad"). InnerVolumeSpecName "kube-api-access-qgmg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:22:26.361429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:26.361398 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ebc545e-7736-47fb-af0d-ab6cfd748fad-maas-api-xks-ca" pod "8ebc545e-7736-47fb-af0d-ab6cfd748fad" (UID: "8ebc545e-7736-47fb-af0d-ab6cfd748fad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:22:26.462391 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:26.462353 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ebc545e-7736-47fb-af0d-ab6cfd748fad-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:22:26.462391 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:26.462391 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgmg7\" (UniqueName: \"kubernetes.io/projected/8ebc545e-7736-47fb-af0d-ab6cfd748fad-kube-api-access-qgmg7\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:22:27.040765 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.040724 4926 generic.go:356] "Generic (PLEG): container finished" podID="d3d50005-aee2-4d9f-b966-7f8a41df5a5a" containerID="0df3de27d0427d80095de282a66195297d01ea84ad9d5dd34536958738e74a1d" exitCode=0 Oct 30 21:22:27.041176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.040782 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" event={"ID":"d3d50005-aee2-4d9f-b966-7f8a41df5a5a","Type":"ContainerDied","Data":"0df3de27d0427d80095de282a66195297d01ea84ad9d5dd34536958738e74a1d"} Oct 30 21:22:27.041356 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.041337 4926 scope.go:122] "RemoveContainer" containerID="0df3de27d0427d80095de282a66195297d01ea84ad9d5dd34536958738e74a1d" Oct 30 21:22:27.043262 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.043232 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:22:27.043368 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.043302 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:22:27.043368 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.043335 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:22:27.044375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.044355 4926 generic.go:356] "Generic (PLEG): container finished" podID="4aca75f2-2a47-484f-b19b-41c6d228cb2c" containerID="dc884dc4bf22f01ff10c78a8c7cf11fde75968bea27bd1aec3e593014052384e" exitCode=0 Oct 30 21:22:27.044468 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.044412 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" event={"ID":"4aca75f2-2a47-484f-b19b-41c6d228cb2c","Type":"ContainerDied","Data":"dc884dc4bf22f01ff10c78a8c7cf11fde75968bea27bd1aec3e593014052384e"} Oct 30 21:22:27.044858 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.044837 4926 scope.go:122] "RemoveContainer" containerID="dc884dc4bf22f01ff10c78a8c7cf11fde75968bea27bd1aec3e593014052384e" Oct 30 21:22:27.050945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.047201 4926 generic.go:356] "Generic (PLEG): container finished" podID="8c6b634e-2a0b-4c0a-aa07-7901e40d77db" containerID="140ab3d0fb774e808bd48d46b3bc9e9165024f96dc38bd16ca136012d9f4c481" exitCode=0 Oct 30 21:22:27.050945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.047313 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" event={"ID":"8c6b634e-2a0b-4c0a-aa07-7901e40d77db","Type":"ContainerDied","Data":"140ab3d0fb774e808bd48d46b3bc9e9165024f96dc38bd16ca136012d9f4c481"} Oct 30 21:22:27.050945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.047390 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:22:27.050945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.047490 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:22:27.050945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.047536 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:22:27.050945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.047685 4926 scope.go:122] "RemoveContainer" containerID="140ab3d0fb774e808bd48d46b3bc9e9165024f96dc38bd16ca136012d9f4c481" Oct 30 21:22:27.050945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.050227 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-s82n5" Oct 30 21:22:27.050945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.050243 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-s82n5" event={"ID":"8ebc545e-7736-47fb-af0d-ab6cfd748fad","Type":"ContainerDied","Data":"3e6ea3dac26c1fdfc6eb98e061146c40632286501b58983815dea162cf2dd84f"} Oct 30 21:22:27.050945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.050278 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e6ea3dac26c1fdfc6eb98e061146c40632286501b58983815dea162cf2dd84f" Oct 30 21:22:27.052743 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.052724 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" event={"ID":"278c1234-a1ec-4205-b4be-b0d567c6cd10","Type":"ContainerStarted","Data":"228edb2f405679d4d16d8959e2efaf079c924cefc538a937c49889272e3eb4fb"} Oct 30 21:22:27.056619 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.056582 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:22:27.056718 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.056670 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:22:27.056776 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.056733 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:22:27.779056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.764698 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:22:27.803057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.784234 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:22:27.803057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.785377 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:22:27.803057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:27.785470 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" podUID="278c1234-a1ec-4205-b4be-b0d567c6cd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 21:22:28.065867 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:28.065770 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" event={"ID":"d3d50005-aee2-4d9f-b966-7f8a41df5a5a","Type":"ContainerStarted","Data":"3c6f0544f64cd19b22520827eb6bb494235131d7ad88350d3a6cf3bb4c09312f"} Oct 30 21:22:28.116163 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:28.116018 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" event={"ID":"4aca75f2-2a47-484f-b19b-41c6d228cb2c","Type":"ContainerStarted","Data":"c2c8aa633a3df83d526b04a2ba33b53738de3747d6880c78ff7612d231275061"} Oct 30 21:22:28.119689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:28.119651 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" event={"ID":"8c6b634e-2a0b-4c0a-aa07-7901e40d77db","Type":"ContainerStarted","Data":"e03d86a9106a4a248fde54f5b8774a3d0760a7099b452ef312a4961a588b7ac6"} Oct 30 21:22:28.385110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:28.379549 4926 patch_prober.go:29] interesting pod/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:22:28.385110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:28.379647 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" podUID="4aca75f2-2a47-484f-b19b-41c6d228cb2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:22:28.385110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:28.380082 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:22:28.414064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:28.390404 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:22:28.533133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:28.524257 4926 patch_prober.go:29] interesting pod/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:22:28.533133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:28.524341 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" podUID="8c6b634e-2a0b-4c0a-aa07-7901e40d77db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:22:28.756130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:28.756090 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:22:28.772245 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:28.772208 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:22:28.796559 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:28.796475 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:22:28.796753 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:28.796583 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" podUID="d3d50005-aee2-4d9f-b966-7f8a41df5a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 21:22:28.796955 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:28.796930 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:22:28.797148 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:28.797120 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" podUID="278c1234-a1ec-4205-b4be-b0d567c6cd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 21:22:29.300184 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:29.300125 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:22:29.308162 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:29.308119 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded" Oct 30 21:22:29.351133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:29.351097 4926 patch_prober.go:29] interesting pod/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:22:29.351440 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:29.351408 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" podUID="4aca75f2-2a47-484f-b19b-41c6d228cb2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:22:29.390472 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:29.390435 4926 patch_prober.go:29] interesting pod/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:22:29.390771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:29.390741 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" podUID="8c6b634e-2a0b-4c0a-aa07-7901e40d77db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:22:29.433256 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:29.433216 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:29.578889 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:29.577880 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:29.599619 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:29.599576 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:29.651983 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:29.651058 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:29.758220 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:29.758163 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:29.766074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:29.765420 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:22:29.766074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:29.765472 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" podUID="d3d50005-aee2-4d9f-b966-7f8a41df5a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 21:22:29.794071 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:29.793335 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:22:29.794071 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:29.793391 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" podUID="278c1234-a1ec-4205-b4be-b0d567c6cd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 21:22:29.846123 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:29.845216 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:30.041677 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.041111 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:30.208057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.207758 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:22:30.208341 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.208163 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded" Oct 30 21:22:30.250054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.239727 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:30.307056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.306474 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:30.307056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.307053 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:22:30.347058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.343492 4926 patch_prober.go:29] interesting pod/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:22:30.347058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.343546 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" podUID="4aca75f2-2a47-484f-b19b-41c6d228cb2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:22:30.429063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.428827 4926 patch_prober.go:29] interesting pod/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:22:30.429063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.428893 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" podUID="8c6b634e-2a0b-4c0a-aa07-7901e40d77db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:22:30.450065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.440102 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:30.450065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.440493 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:22:30.576317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.502126 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:22:30.576317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.502558 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:30.599010 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.595122 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:30.599010 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.595567 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:22:30.630264 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.600098 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:30.630264 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.600450 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:22:30.758846 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.758713 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:30.760080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.760052 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:22:30.763878 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.762888 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:22:30.764259 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.764231 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" podUID="d3d50005-aee2-4d9f-b966-7f8a41df5a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 21:22:30.775423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.775371 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:22:30.775578 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.775427 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" podUID="278c1234-a1ec-4205-b4be-b0d567c6cd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 21:22:30.854441 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:30.854343 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:31.167852 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.154851 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:31.221057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.196165 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:22:31.536109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.211141 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:31.577103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.239858 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:31.603308 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.334127 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:31.627095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.612111 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 21:22:31.627095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.360268 4926 patch_prober.go:29] interesting pod/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:22:31.627095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.612464 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" podUID="4aca75f2-2a47-484f-b19b-41c6d228cb2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:22:31.627095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.528897 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:31.627095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.619213 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 21:22:31.627095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.528964 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 21:22:31.627095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.403124 4926 patch_prober.go:29] interesting pod/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:22:31.627095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.624693 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" podUID="8c6b634e-2a0b-4c0a-aa07-7901e40d77db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:22:31.627095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.449389 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:31.628403 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.334172 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:31.650416 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.528944 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:22:31.658626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.658587 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:31.802827 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.802108 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": context deadline exceeded" Oct 30 21:22:31.802827 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.802216 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:22:31.818226 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.807884 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:22:31.818226 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.807931 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" podUID="d3d50005-aee2-4d9f-b966-7f8a41df5a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 21:22:31.818226 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.808698 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:22:31.818226 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.808735 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" podUID="278c1234-a1ec-4205-b4be-b0d567c6cd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 21:22:31.828117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.827692 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:22:31.828117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.827745 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:22:31.828117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.827780 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:22:31.828117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.827813 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:22:31.828117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.827845 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:22:31.847044 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.846998 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:22:31.864333 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:31.864277 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:32.216057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.210287 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:32.216542 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.216506 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Oct 30 21:22:32.272153 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.272110 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded" Oct 30 21:22:32.555266 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.555168 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded" Oct 30 21:22:32.581053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.579646 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:22:32.588186 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.585482 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:22:32.588186 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.587741 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:32.624178 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.624145 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:22:32.641346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.641296 4926 patch_prober.go:29] interesting pod/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:22:32.725056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.719597 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" podUID="8c6b634e-2a0b-4c0a-aa07-7901e40d77db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:22:32.725056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.645182 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded" Oct 30 21:22:32.725056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.645695 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 21:22:32.764130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.645114 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 21:22:32.787966 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.642069 4926 patch_prober.go:29] interesting pod/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:22:32.787966 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.775274 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" podUID="4aca75f2-2a47-484f-b19b-41c6d228cb2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:22:32.787966 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.646148 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 21:22:32.787966 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.646066 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:32.787966 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.646092 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 21:22:32.787966 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.646125 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:22:32.787966 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.646347 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 21:22:32.787966 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.776788 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:22:32.790329 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.694148 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 21:22:32.791962 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.791930 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:22:32.798845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.798807 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded" Oct 30 21:22:32.887998 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.821239 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:22:32.888343 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.888300 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" podUID="d3d50005-aee2-4d9f-b966-7f8a41df5a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 21:22:32.888504 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.829666 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:22:32.888665 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.888652 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:22:32.888839 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.828977 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:32.889073 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.826969 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:22:32.889222 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.889196 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" podUID="278c1234-a1ec-4205-b4be-b0d567c6cd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 21:22:32.889447 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.889432 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:22:32.950780 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:32.950139 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:33.282289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:33.282246 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:33.422206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:33.422170 4926 patch_prober.go:29] interesting pod/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:22:33.422527 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:33.422493 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" podUID="4aca75f2-2a47-484f-b19b-41c6d228cb2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:22:33.472069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:33.462167 4926 patch_prober.go:29] interesting pod/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:22:33.472326 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:33.472293 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" podUID="8c6b634e-2a0b-4c0a-aa07-7901e40d77db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:22:33.545860 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:33.545773 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 21:22:33.626078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:33.609315 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:33.762065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:33.751292 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 21:22:33.774877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:33.774832 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:33.776323 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:33.776289 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:33.777520 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:33.777493 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:33.778709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:33.778681 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:33.779938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:33.779912 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:33.831282 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:33.831190 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:22:33.831570 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:33.831520 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" podUID="d3d50005-aee2-4d9f-b966-7f8a41df5a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 21:22:33.890429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:33.849531 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:22:33.911213 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:33.911171 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" podUID="278c1234-a1ec-4205-b4be-b0d567c6cd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 21:22:33.911507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:33.858690 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:22:33.911985 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:33.858730 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:33.915573 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:33.891200 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:33.931065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:33.870109 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:33.999789 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:33.999745 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:34.294077 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:34.283589 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 21:22:34.342738 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:34.342706 4926 patch_prober.go:29] interesting pod/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:22:34.343066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:34.343016 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" podUID="4aca75f2-2a47-484f-b19b-41c6d228cb2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:22:34.395559 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:34.395525 4926 patch_prober.go:29] interesting pod/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:22:34.395795 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:34.395770 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" podUID="8c6b634e-2a0b-4c0a-aa07-7901e40d77db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:22:34.448053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:34.445138 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:22:34.494432 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:34.494171 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:34.724059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:34.722409 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:22:34.771054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:34.770825 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:22:34.772283 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:34.772248 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:22:34.772428 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:34.772305 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" podUID="278c1234-a1ec-4205-b4be-b0d567c6cd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 21:22:34.772563 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:34.772543 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:22:34.772702 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:34.772682 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" podUID="d3d50005-aee2-4d9f-b966-7f8a41df5a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 21:22:34.925054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:34.772683 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:34.925054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:34.772256 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 21:22:34.925054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:34.779311 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:34.925054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:34.779337 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 21:22:34.925054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:34.779356 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:22:34.925054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:34.779378 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 30 21:22:34.938136 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:34.841196 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:34.938136 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:34.936122 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:34.938136 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:34.936615 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:22:35.001502 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:35.001380 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:35.285131 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:35.285028 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:22:35.285419 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:35.285403 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:22:35.318857 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:35.318816 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:35.369710 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:35.369677 4926 patch_prober.go:29] interesting pod/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:22:35.379304 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:35.379265 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" podUID="4aca75f2-2a47-484f-b19b-41c6d228cb2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:22:35.392428 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:35.392390 4926 patch_prober.go:29] interesting pod/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:22:35.392685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:35.392659 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" podUID="8c6b634e-2a0b-4c0a-aa07-7901e40d77db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:22:35.662196 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:35.644403 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:22:35.734056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:35.729804 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:35.836133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:35.834201 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 21:22:35.836133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:35.834279 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" podUID="278c1234-a1ec-4205-b4be-b0d567c6cd10" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 21:22:35.836133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:35.835896 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:22:35.836133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:35.835944 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" podUID="d3d50005-aee2-4d9f-b966-7f8a41df5a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 21:22:35.969083 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:35.938382 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:22:35.969083 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:35.964444 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:22:36.016769 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:35.988925 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 21:22:36.046619 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:36.046558 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:36.047664 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:36.047615 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:22:36.050310 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:36.050270 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 21:22:36.097572 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:36.097542 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:22:36.256363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:36.256236 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:36.410103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:36.384127 4926 patch_prober.go:29] interesting pod/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:22:36.410465 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:36.410421 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" podUID="4aca75f2-2a47-484f-b19b-41c6d228cb2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:22:36.411657 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:36.411623 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:36.414815 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:36.414790 4926 patch_prober.go:29] interesting pod/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:22:36.414993 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:36.414972 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" podUID="8c6b634e-2a0b-4c0a-aa07-7901e40d77db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:22:36.554119 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:36.553151 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded" Oct 30 21:22:36.611164 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:36.611119 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded" Oct 30 21:22:36.658223 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:36.658170 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8k468" podUID="97d86e50-7f90-4650-bef4-2a4cd91c1d61" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.37:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:36.807411 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:36.774353 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:36.847087 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:36.833497 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:22:36.853128 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:36.852664 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Oct 30 21:22:36.853128 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:36.852724 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" podUID="d3d50005-aee2-4d9f-b966-7f8a41df5a5a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Oct 30 21:22:36.914416 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:36.914385 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:22:36.933051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:36.932999 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:22:37.146114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:37.064592 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:37.146114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:37.066401 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:37.146114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:37.095637 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:37.146114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:37.104466 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:37.146114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:37.121141 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:37.151062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:37.146611 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:37.315190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:37.315138 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:37.346054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:37.346002 4926 patch_prober.go:29] interesting pod/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body= Oct 30 21:22:37.346317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:37.346288 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" podUID="4aca75f2-2a47-484f-b19b-41c6d228cb2c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" Oct 30 21:22:37.363554 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:37.363526 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:22:37.394133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:37.394098 4926 patch_prober.go:29] interesting pod/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Oct 30 21:22:37.397205 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:37.397105 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" podUID="8c6b634e-2a0b-4c0a-aa07-7901e40d77db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Oct 30 21:22:37.539104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:37.523865 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-8k468" podUID="97d86e50-7f90-4650-bef4-2a4cd91c1d61" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.37:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:37.550638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:37.550149 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:22:37.597672 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:37.596355 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:22:37.602133 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:22:37.602055 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:22:37.723957 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:37.723927 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:22:37.880785 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:37.879458 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:22:37.891988 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:37.891746 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:22:38.346411 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:38.346374 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:22:38.346873 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:38.346858 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:22:38.347677 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:38.347654 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:22:38.395463 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:38.395431 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:22:38.396691 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:38.396670 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:22:52.273940 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:52.273905 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:22:52.565401 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:52.565316 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw"] Oct 30 21:22:52.566188 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:52.566166 4926 cpu_manager.go:414] "containerMap: removing container" podUID="8ebc545e-7736-47fb-af0d-ab6cfd748fad" containerName="revoke-keys" Oct 30 21:22:52.566188 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:52.566186 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ebc545e-7736-47fb-af0d-ab6cfd748fad" containerName="revoke-keys" Oct 30 21:22:52.566388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:52.566326 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ebc545e-7736-47fb-af0d-ab6cfd748fad" containerName="revoke-keys" Oct 30 21:22:52.569268 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:52.569246 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw" Oct 30 21:22:52.588856 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:52.588829 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw"] Oct 30 21:22:52.596126 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:52.596092 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:22:52.596271 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:52.596243 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:22:52.721263 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:52.721224 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj9hf\" (UniqueName: \"kubernetes.io/projected/f8da07db-83fb-49ad-950c-9206d47c0bec-kube-api-access-nj9hf\") pod \"maas-api-revoke-sub-a814527b-sbwlw\" (UID: \"f8da07db-83fb-49ad-950c-9206d47c0bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw" Oct 30 21:22:52.721466 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:52.721419 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8da07db-83fb-49ad-950c-9206d47c0bec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a814527b-sbwlw\" (UID: \"f8da07db-83fb-49ad-950c-9206d47c0bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw" Oct 30 21:22:52.721612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:52.721596 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8da07db-83fb-49ad-950c-9206d47c0bec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a814527b-sbwlw\" (UID: \"f8da07db-83fb-49ad-950c-9206d47c0bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw" Oct 30 21:22:52.823520 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:52.823429 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj9hf\" (UniqueName: \"kubernetes.io/projected/f8da07db-83fb-49ad-950c-9206d47c0bec-kube-api-access-nj9hf\") pod \"maas-api-revoke-sub-a814527b-sbwlw\" (UID: \"f8da07db-83fb-49ad-950c-9206d47c0bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw" Oct 30 21:22:52.823699 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:52.823610 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8da07db-83fb-49ad-950c-9206d47c0bec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a814527b-sbwlw\" (UID: \"f8da07db-83fb-49ad-950c-9206d47c0bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw" Oct 30 21:22:52.823829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:52.823803 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8da07db-83fb-49ad-950c-9206d47c0bec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a814527b-sbwlw\" (UID: \"f8da07db-83fb-49ad-950c-9206d47c0bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw" Oct 30 21:22:52.824964 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:52.824710 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8da07db-83fb-49ad-950c-9206d47c0bec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a814527b-sbwlw\" (UID: \"f8da07db-83fb-49ad-950c-9206d47c0bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw" Oct 30 21:22:52.826774 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:52.826754 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8da07db-83fb-49ad-950c-9206d47c0bec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a814527b-sbwlw\" (UID: \"f8da07db-83fb-49ad-950c-9206d47c0bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw" Oct 30 21:22:52.842272 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:52.842244 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj9hf\" (UniqueName: \"kubernetes.io/projected/f8da07db-83fb-49ad-950c-9206d47c0bec-kube-api-access-nj9hf\") pod \"maas-api-revoke-sub-a814527b-sbwlw\" (UID: \"f8da07db-83fb-49ad-950c-9206d47c0bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw" Oct 30 21:22:52.886104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:52.886068 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw" Oct 30 21:22:53.209998 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:53.209967 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw"] Oct 30 21:22:53.211094 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:22:53.211068 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8da07db_83fb_49ad_950c_9206d47c0bec.slice/crio-aac64ba8877caac9001f0348258697d132bb19bdf5d90ff71458acdc7c03e898 WatchSource:0}: Error finding container aac64ba8877caac9001f0348258697d132bb19bdf5d90ff71458acdc7c03e898: Status 404 returned error can't find the container with id aac64ba8877caac9001f0348258697d132bb19bdf5d90ff71458acdc7c03e898 Oct 30 21:22:54.220073 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:54.219288 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw" event={"ID":"f8da07db-83fb-49ad-950c-9206d47c0bec","Type":"ContainerStarted","Data":"17d8f8263b8c94e9002ca4edf4792c100c0dabd4e6f10b6bf5c3411699ef84b8"} Oct 30 21:22:54.220073 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:54.219359 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw" event={"ID":"f8da07db-83fb-49ad-950c-9206d47c0bec","Type":"ContainerStarted","Data":"aac64ba8877caac9001f0348258697d132bb19bdf5d90ff71458acdc7c03e898"} Oct 30 21:22:54.305058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:54.296832 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw" podStartSLOduration=2.296810748 podStartE2EDuration="2.296810748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:22:54.266468458 +0000 UTC m=+1361.387542206" watchObservedRunningTime="2026-10-30 21:22:54.296810748 +0000 UTC m=+1361.417884496" Oct 30 21:22:55.253412 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:55.253213 4926 generic.go:356] "Generic (PLEG): container finished" podID="f8da07db-83fb-49ad-950c-9206d47c0bec" containerID="17d8f8263b8c94e9002ca4edf4792c100c0dabd4e6f10b6bf5c3411699ef84b8" exitCode=0 Oct 30 21:22:55.253806 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:55.253581 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw" event={"ID":"f8da07db-83fb-49ad-950c-9206d47c0bec","Type":"ContainerDied","Data":"17d8f8263b8c94e9002ca4edf4792c100c0dabd4e6f10b6bf5c3411699ef84b8"} Oct 30 21:22:56.804501 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:56.804231 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw" Oct 30 21:22:56.991553 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:56.991402 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8da07db-83fb-49ad-950c-9206d47c0bec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8da07db-83fb-49ad-950c-9206d47c0bec-maas-api-xks-ca\") pod \"f8da07db-83fb-49ad-950c-9206d47c0bec\" (UID: \"f8da07db-83fb-49ad-950c-9206d47c0bec\") " Oct 30 21:22:56.991553 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:56.991537 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8da07db-83fb-49ad-950c-9206d47c0bec-kube-api-access-nj9hf\" (UniqueName: \"kubernetes.io/projected/f8da07db-83fb-49ad-950c-9206d47c0bec-kube-api-access-nj9hf\") pod \"f8da07db-83fb-49ad-950c-9206d47c0bec\" (UID: \"f8da07db-83fb-49ad-950c-9206d47c0bec\") " Oct 30 21:22:56.993451 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:56.993420 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8da07db-83fb-49ad-950c-9206d47c0bec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8da07db-83fb-49ad-950c-9206d47c0bec-maas-api-openshift-ca\") pod \"f8da07db-83fb-49ad-950c-9206d47c0bec\" (UID: \"f8da07db-83fb-49ad-950c-9206d47c0bec\") " Oct 30 21:22:56.993866 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:56.993841 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8da07db-83fb-49ad-950c-9206d47c0bec-maas-api-openshift-ca" pod "f8da07db-83fb-49ad-950c-9206d47c0bec" (UID: "f8da07db-83fb-49ad-950c-9206d47c0bec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:22:56.997085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:56.994881 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8da07db-83fb-49ad-950c-9206d47c0bec-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:22:57.003685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:56.997628 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8da07db-83fb-49ad-950c-9206d47c0bec-kube-api-access-nj9hf" pod "f8da07db-83fb-49ad-950c-9206d47c0bec" (UID: "f8da07db-83fb-49ad-950c-9206d47c0bec"). InnerVolumeSpecName "kube-api-access-nj9hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:22:57.032875 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:57.032840 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8da07db-83fb-49ad-950c-9206d47c0bec-maas-api-xks-ca" pod "f8da07db-83fb-49ad-950c-9206d47c0bec" (UID: "f8da07db-83fb-49ad-950c-9206d47c0bec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:22:57.112406 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:57.112361 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8da07db-83fb-49ad-950c-9206d47c0bec-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:22:57.112406 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:57.112402 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj9hf\" (UniqueName: \"kubernetes.io/projected/f8da07db-83fb-49ad-950c-9206d47c0bec-kube-api-access-nj9hf\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:22:57.271739 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:57.271657 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw" Oct 30 21:22:57.271739 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:57.271694 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw" event={"ID":"f8da07db-83fb-49ad-950c-9206d47c0bec","Type":"ContainerDied","Data":"aac64ba8877caac9001f0348258697d132bb19bdf5d90ff71458acdc7c03e898"} Oct 30 21:22:57.271960 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:22:57.271758 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aac64ba8877caac9001f0348258697d132bb19bdf5d90ff71458acdc7c03e898" Oct 30 21:23:00.754655 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:00.754620 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm"] Oct 30 21:23:00.755066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:00.754810 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" podUID="5397fa31-83d0-4500-94bb-16e1848efe2f" containerName="istio-proxy" containerID="cri-o://8f4150dc8404f0a64af3b1ab3e55246bffbfcf2b27670a8c6241b501b063ce4d" gracePeriod=30 Oct 30 21:23:02.138546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:02.138509 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj"] Oct 30 21:23:02.139000 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:02.138716 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" podUID="37bf8b31-c305-4605-b262-f7f0aa6d2644" containerName="istio-proxy" containerID="cri-o://7fe18c40a71b364d1f49be81f1cc870d034725b5a29b299bde30419e5e8af284" gracePeriod=30 Oct 30 21:23:02.196056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:02.195673 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7"] Oct 30 21:23:02.200895 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:02.198227 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" podUID="83749c42-ef6a-4755-acbf-2166a604a6e6" containerName="istio-proxy" containerID="cri-o://aeb066d5d6f84bcd64676d129486de4ca6d8acc4ce81f3031664e5414d943cde" gracePeriod=30 Oct 30 21:23:02.263819 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:02.263136 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp"] Oct 30 21:23:02.263819 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:02.263328 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" podUID="3f026f40-c790-4639-8975-98e56da80ce9" containerName="istio-proxy" containerID="cri-o://4c721600dc428d4b2935c3d0fb16dce12e3a367fe25151615922d49e6296720d" gracePeriod=30 Oct 30 21:23:02.270130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:02.270099 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64"] Oct 30 21:23:02.270668 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:02.270376 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" podUID="b42d62ed-b880-4a25-978c-47c703a5cdd3" containerName="istio-proxy" containerID="cri-o://eb143e0dcc55a237f94e1192b7449d8f1b34735672f2e91f4a70249e5d6cae62" gracePeriod=30 Oct 30 21:23:02.888207 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:02.888160 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9"] Oct 30 21:23:02.888474 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:02.888447 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" podUID="a3061a9a-e0c1-46c9-914b-a5aba158ec6a" containerName="istio-proxy" containerID="cri-o://6f0b7410464cb92dda1adffffb7ccf1e766b0a966d29d90b04e90754fb97bb65" gracePeriod=30 Oct 30 21:23:02.893373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:02.893337 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9"] Oct 30 21:23:02.893582 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:02.893561 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" podUID="8a80b31b-f4f7-47bc-abe7-27b0505919f5" containerName="istio-proxy" containerID="cri-o://7ef02ebb7fa0b590d8c6acd6c7ae552e30b1c1c8d62e7547cf1a0baf65de5de8" gracePeriod=30 Oct 30 21:23:06.056661 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.056631 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:23:06.186606 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.186567 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5397fa31-83d0-4500-94bb-16e1848efe2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5397fa31-83d0-4500-94bb-16e1848efe2f-istiod-ca-cert\") pod \"5397fa31-83d0-4500-94bb-16e1848efe2f\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " Oct 30 21:23:06.186775 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.186613 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-token\" (UniqueName: \"kubernetes.io/projected/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-token\") pod \"5397fa31-83d0-4500-94bb-16e1848efe2f\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " Oct 30 21:23:06.186775 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.186646 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-credential-socket\") pod \"5397fa31-83d0-4500-94bb-16e1848efe2f\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " Oct 30 21:23:06.186775 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.186675 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-workload-socket\") pod \"5397fa31-83d0-4500-94bb-16e1848efe2f\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " Oct 30 21:23:06.186966 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.186807 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-envoy\") pod \"5397fa31-83d0-4500-94bb-16e1848efe2f\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " Oct 30 21:23:06.187076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.186940 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-credential-socket" pod "5397fa31-83d0-4500-94bb-16e1848efe2f" (UID: "5397fa31-83d0-4500-94bb-16e1848efe2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:06.187076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.187023 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-workload-socket" pod "5397fa31-83d0-4500-94bb-16e1848efe2f" (UID: "5397fa31-83d0-4500-94bb-16e1848efe2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:06.187280 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.187258 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-workload-certs\") pod \"5397fa31-83d0-4500-94bb-16e1848efe2f\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " Oct 30 21:23:06.187443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.187420 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-data\") pod \"5397fa31-83d0-4500-94bb-16e1848efe2f\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " Oct 30 21:23:06.187563 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.187321 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5397fa31-83d0-4500-94bb-16e1848efe2f-istiod-ca-cert" pod "5397fa31-83d0-4500-94bb-16e1848efe2f" (UID: "5397fa31-83d0-4500-94bb-16e1848efe2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:06.187563 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.187470 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-workload-certs" pod "5397fa31-83d0-4500-94bb-16e1848efe2f" (UID: "5397fa31-83d0-4500-94bb-16e1848efe2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:06.187563 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.187529 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-podinfo\") pod \"5397fa31-83d0-4500-94bb-16e1848efe2f\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " Oct 30 21:23:06.187730 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.187571 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5397fa31-83d0-4500-94bb-16e1848efe2f-kube-api-access-9895q\" (UniqueName: \"kubernetes.io/projected/5397fa31-83d0-4500-94bb-16e1848efe2f-kube-api-access-9895q\") pod \"5397fa31-83d0-4500-94bb-16e1848efe2f\" (UID: \"5397fa31-83d0-4500-94bb-16e1848efe2f\") " Oct 30 21:23:06.187786 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.187750 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-data" pod "5397fa31-83d0-4500-94bb-16e1848efe2f" (UID: "5397fa31-83d0-4500-94bb-16e1848efe2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:06.188588 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.188561 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5397fa31-83d0-4500-94bb-16e1848efe2f-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:06.188708 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.188593 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:06.188708 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.188614 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:06.188708 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.188630 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:06.188708 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.188641 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:06.189243 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.189218 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-token" pod "5397fa31-83d0-4500-94bb-16e1848efe2f" (UID: "5397fa31-83d0-4500-94bb-16e1848efe2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:06.189409 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.189391 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-envoy" pod "5397fa31-83d0-4500-94bb-16e1848efe2f" (UID: "5397fa31-83d0-4500-94bb-16e1848efe2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:06.189825 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.189808 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-podinfo" pod "5397fa31-83d0-4500-94bb-16e1848efe2f" (UID: "5397fa31-83d0-4500-94bb-16e1848efe2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:23:06.190307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.190287 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5397fa31-83d0-4500-94bb-16e1848efe2f-kube-api-access-9895q" pod "5397fa31-83d0-4500-94bb-16e1848efe2f" (UID: "5397fa31-83d0-4500-94bb-16e1848efe2f"). InnerVolumeSpecName "kube-api-access-9895q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:06.290750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.290669 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:06.290750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.290700 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:06.290750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.290708 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5397fa31-83d0-4500-94bb-16e1848efe2f-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:06.290750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.290718 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9895q\" (UniqueName: \"kubernetes.io/projected/5397fa31-83d0-4500-94bb-16e1848efe2f-kube-api-access-9895q\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:06.361377 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.361342 4926 generic.go:356] "Generic (PLEG): container finished" podID="5397fa31-83d0-4500-94bb-16e1848efe2f" containerID="8f4150dc8404f0a64af3b1ab3e55246bffbfcf2b27670a8c6241b501b063ce4d" exitCode=0 Oct 30 21:23:06.361546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.361396 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" Oct 30 21:23:06.361546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.361396 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" event={"ID":"5397fa31-83d0-4500-94bb-16e1848efe2f","Type":"ContainerDied","Data":"8f4150dc8404f0a64af3b1ab3e55246bffbfcf2b27670a8c6241b501b063ce4d"} Oct 30 21:23:06.361546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.361539 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm" event={"ID":"5397fa31-83d0-4500-94bb-16e1848efe2f","Type":"ContainerDied","Data":"5aa4eedd22596ebba2d1ba747d8b44c2a08c606a66622df46dab325cc33e0f89"} Oct 30 21:23:06.361652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.361568 4926 scope.go:122] "RemoveContainer" containerID="8f4150dc8404f0a64af3b1ab3e55246bffbfcf2b27670a8c6241b501b063ce4d" Oct 30 21:23:06.374118 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.374097 4926 scope.go:122] "RemoveContainer" containerID="8f4150dc8404f0a64af3b1ab3e55246bffbfcf2b27670a8c6241b501b063ce4d" Oct 30 21:23:06.374905 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:23:06.374876 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f4150dc8404f0a64af3b1ab3e55246bffbfcf2b27670a8c6241b501b063ce4d\": container with ID starting with 8f4150dc8404f0a64af3b1ab3e55246bffbfcf2b27670a8c6241b501b063ce4d not found: ID does not exist" containerID="8f4150dc8404f0a64af3b1ab3e55246bffbfcf2b27670a8c6241b501b063ce4d" Oct 30 21:23:06.375018 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.374907 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f4150dc8404f0a64af3b1ab3e55246bffbfcf2b27670a8c6241b501b063ce4d"} err="failed to get container status \"8f4150dc8404f0a64af3b1ab3e55246bffbfcf2b27670a8c6241b501b063ce4d\": rpc error: code = NotFound desc = could not find container \"8f4150dc8404f0a64af3b1ab3e55246bffbfcf2b27670a8c6241b501b063ce4d\": container with ID starting with 8f4150dc8404f0a64af3b1ab3e55246bffbfcf2b27670a8c6241b501b063ce4d not found: ID does not exist" Oct 30 21:23:06.414259 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.414228 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm"] Oct 30 21:23:06.425376 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:06.425346 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-h4twm"] Oct 30 21:23:07.020220 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.020189 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5397fa31-83d0-4500-94bb-16e1848efe2f" path="/var/lib/kubelet/pods/5397fa31-83d0-4500-94bb-16e1848efe2f/volumes" Oct 30 21:23:07.374427 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.374393 4926 generic.go:356] "Generic (PLEG): container finished" podID="83749c42-ef6a-4755-acbf-2166a604a6e6" containerID="aeb066d5d6f84bcd64676d129486de4ca6d8acc4ce81f3031664e5414d943cde" exitCode=0 Oct 30 21:23:07.374881 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.374465 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" event={"ID":"83749c42-ef6a-4755-acbf-2166a604a6e6","Type":"ContainerDied","Data":"aeb066d5d6f84bcd64676d129486de4ca6d8acc4ce81f3031664e5414d943cde"} Oct 30 21:23:07.378495 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.378471 4926 generic.go:356] "Generic (PLEG): container finished" podID="37bf8b31-c305-4605-b262-f7f0aa6d2644" containerID="7fe18c40a71b364d1f49be81f1cc870d034725b5a29b299bde30419e5e8af284" exitCode=0 Oct 30 21:23:07.378617 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.378523 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" event={"ID":"37bf8b31-c305-4605-b262-f7f0aa6d2644","Type":"ContainerDied","Data":"7fe18c40a71b364d1f49be81f1cc870d034725b5a29b299bde30419e5e8af284"} Oct 30 21:23:07.527950 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:23:07.527827 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:23:07.560519 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.560452 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:23:07.662509 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.662119 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:23:07.745056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.744024 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37bf8b31-c305-4605-b262-f7f0aa6d2644-kube-api-access-tqbjn\" (UniqueName: \"kubernetes.io/projected/37bf8b31-c305-4605-b262-f7f0aa6d2644-kube-api-access-tqbjn\") pod \"37bf8b31-c305-4605-b262-f7f0aa6d2644\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " Oct 30 21:23:07.745056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.744130 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-envoy\") pod \"83749c42-ef6a-4755-acbf-2166a604a6e6\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " Oct 30 21:23:07.745056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.744206 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37bf8b31-c305-4605-b262-f7f0aa6d2644-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37bf8b31-c305-4605-b262-f7f0aa6d2644-istiod-ca-cert\") pod \"37bf8b31-c305-4605-b262-f7f0aa6d2644\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " Oct 30 21:23:07.745056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.744277 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-workload-socket\") pod \"83749c42-ef6a-4755-acbf-2166a604a6e6\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " Oct 30 21:23:07.745056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.744312 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-envoy\") pod \"37bf8b31-c305-4605-b262-f7f0aa6d2644\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " Oct 30 21:23:07.745056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.744352 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83749c42-ef6a-4755-acbf-2166a604a6e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83749c42-ef6a-4755-acbf-2166a604a6e6-istiod-ca-cert\") pod \"83749c42-ef6a-4755-acbf-2166a604a6e6\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " Oct 30 21:23:07.745056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.744430 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-workload-certs\") pod \"83749c42-ef6a-4755-acbf-2166a604a6e6\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " Oct 30 21:23:07.745056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.744498 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-token\" (UniqueName: \"kubernetes.io/projected/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-token\") pod \"83749c42-ef6a-4755-acbf-2166a604a6e6\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " Oct 30 21:23:07.745056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.744539 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-podinfo\") pod \"37bf8b31-c305-4605-b262-f7f0aa6d2644\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " Oct 30 21:23:07.745056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.744576 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-data\") pod \"83749c42-ef6a-4755-acbf-2166a604a6e6\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " Oct 30 21:23:07.745056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.744604 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-workload-certs\") pod \"37bf8b31-c305-4605-b262-f7f0aa6d2644\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " Oct 30 21:23:07.745056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.744661 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-credential-socket\") pod \"37bf8b31-c305-4605-b262-f7f0aa6d2644\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " Oct 30 21:23:07.745056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.744764 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-data\") pod \"37bf8b31-c305-4605-b262-f7f0aa6d2644\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " Oct 30 21:23:07.745056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.744836 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-podinfo\") pod \"83749c42-ef6a-4755-acbf-2166a604a6e6\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " Oct 30 21:23:07.745056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.744871 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-credential-socket\") pod \"83749c42-ef6a-4755-acbf-2166a604a6e6\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " Oct 30 21:23:07.745056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.744924 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-token\" (UniqueName: \"kubernetes.io/projected/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-token\") pod \"37bf8b31-c305-4605-b262-f7f0aa6d2644\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " Oct 30 21:23:07.745056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.744964 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83749c42-ef6a-4755-acbf-2166a604a6e6-kube-api-access-rgn7q\" (UniqueName: \"kubernetes.io/projected/83749c42-ef6a-4755-acbf-2166a604a6e6-kube-api-access-rgn7q\") pod \"83749c42-ef6a-4755-acbf-2166a604a6e6\" (UID: \"83749c42-ef6a-4755-acbf-2166a604a6e6\") " Oct 30 21:23:07.745056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.745010 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-workload-socket\") pod \"37bf8b31-c305-4605-b262-f7f0aa6d2644\" (UID: \"37bf8b31-c305-4605-b262-f7f0aa6d2644\") " Oct 30 21:23:07.750681 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.750640 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-workload-socket" pod "37bf8b31-c305-4605-b262-f7f0aa6d2644" (UID: "37bf8b31-c305-4605-b262-f7f0aa6d2644"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:07.752780 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.752499 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37bf8b31-c305-4605-b262-f7f0aa6d2644-istiod-ca-cert" pod "37bf8b31-c305-4605-b262-f7f0aa6d2644" (UID: "37bf8b31-c305-4605-b262-f7f0aa6d2644"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:07.753012 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.752886 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-data" pod "37bf8b31-c305-4605-b262-f7f0aa6d2644" (UID: "37bf8b31-c305-4605-b262-f7f0aa6d2644"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:07.753204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.753140 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-workload-certs" pod "83749c42-ef6a-4755-acbf-2166a604a6e6" (UID: "83749c42-ef6a-4755-acbf-2166a604a6e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:07.753468 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.753446 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-data" pod "83749c42-ef6a-4755-acbf-2166a604a6e6" (UID: "83749c42-ef6a-4755-acbf-2166a604a6e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:07.753761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.753737 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83749c42-ef6a-4755-acbf-2166a604a6e6-istiod-ca-cert" pod "83749c42-ef6a-4755-acbf-2166a604a6e6" (UID: "83749c42-ef6a-4755-acbf-2166a604a6e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:07.757824 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.755393 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-workload-certs" pod "37bf8b31-c305-4605-b262-f7f0aa6d2644" (UID: "37bf8b31-c305-4605-b262-f7f0aa6d2644"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:07.757824 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.755635 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-credential-socket" pod "37bf8b31-c305-4605-b262-f7f0aa6d2644" (UID: "37bf8b31-c305-4605-b262-f7f0aa6d2644"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:07.757824 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.755949 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-workload-socket" pod "83749c42-ef6a-4755-acbf-2166a604a6e6" (UID: "83749c42-ef6a-4755-acbf-2166a604a6e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:07.757824 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.755962 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-credential-socket" pod "83749c42-ef6a-4755-acbf-2166a604a6e6" (UID: "83749c42-ef6a-4755-acbf-2166a604a6e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:07.757824 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.757223 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-envoy" pod "83749c42-ef6a-4755-acbf-2166a604a6e6" (UID: "83749c42-ef6a-4755-acbf-2166a604a6e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:07.762027 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.761977 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-token" pod "37bf8b31-c305-4605-b262-f7f0aa6d2644" (UID: "37bf8b31-c305-4605-b262-f7f0aa6d2644"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:07.770274 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.767728 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37bf8b31-c305-4605-b262-f7f0aa6d2644-kube-api-access-tqbjn" pod "37bf8b31-c305-4605-b262-f7f0aa6d2644" (UID: "37bf8b31-c305-4605-b262-f7f0aa6d2644"). InnerVolumeSpecName "kube-api-access-tqbjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:07.770274 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.768242 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-envoy" pod "37bf8b31-c305-4605-b262-f7f0aa6d2644" (UID: "37bf8b31-c305-4605-b262-f7f0aa6d2644"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:07.770274 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.769099 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-podinfo" pod "37bf8b31-c305-4605-b262-f7f0aa6d2644" (UID: "37bf8b31-c305-4605-b262-f7f0aa6d2644"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:23:07.770274 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.769220 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83749c42-ef6a-4755-acbf-2166a604a6e6-kube-api-access-rgn7q" pod "83749c42-ef6a-4755-acbf-2166a604a6e6" (UID: "83749c42-ef6a-4755-acbf-2166a604a6e6"). InnerVolumeSpecName "kube-api-access-rgn7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:07.770274 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.769305 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-podinfo" pod "83749c42-ef6a-4755-acbf-2166a604a6e6" (UID: "83749c42-ef6a-4755-acbf-2166a604a6e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:23:07.770274 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.770167 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-token" pod "83749c42-ef6a-4755-acbf-2166a604a6e6" (UID: "83749c42-ef6a-4755-acbf-2166a604a6e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:07.798008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.797543 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:23:07.799007 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.798983 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:23:07.846689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.846606 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f026f40-c790-4639-8975-98e56da80ce9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f026f40-c790-4639-8975-98e56da80ce9-istio-podinfo\") pod \"3f026f40-c790-4639-8975-98e56da80ce9\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " Oct 30 21:23:07.846689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.846661 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-istio-envoy\") pod \"3f026f40-c790-4639-8975-98e56da80ce9\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " Oct 30 21:23:07.846897 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.846724 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-workload-certs\") pod \"b42d62ed-b880-4a25-978c-47c703a5cdd3\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " Oct 30 21:23:07.846897 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.846752 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b42d62ed-b880-4a25-978c-47c703a5cdd3-kube-api-access-52pbj\" (UniqueName: \"kubernetes.io/projected/b42d62ed-b880-4a25-978c-47c703a5cdd3-kube-api-access-52pbj\") pod \"b42d62ed-b880-4a25-978c-47c703a5cdd3\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " Oct 30 21:23:07.846897 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.846780 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-podinfo\") pod \"b42d62ed-b880-4a25-978c-47c703a5cdd3\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " Oct 30 21:23:07.846897 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.846806 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-workload-certs\") pod \"3f026f40-c790-4639-8975-98e56da80ce9\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " Oct 30 21:23:07.846897 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.846828 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-istio-data\") pod \"3f026f40-c790-4639-8975-98e56da80ce9\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " Oct 30 21:23:07.846897 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.846884 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-token\" (UniqueName: \"kubernetes.io/projected/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-token\") pod \"b42d62ed-b880-4a25-978c-47c703a5cdd3\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " Oct 30 21:23:07.847203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.846916 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-envoy\") pod \"b42d62ed-b880-4a25-978c-47c703a5cdd3\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " Oct 30 21:23:07.847203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.846972 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b42d62ed-b880-4a25-978c-47c703a5cdd3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42d62ed-b880-4a25-978c-47c703a5cdd3-istiod-ca-cert\") pod \"b42d62ed-b880-4a25-978c-47c703a5cdd3\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " Oct 30 21:23:07.847203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.847079 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f026f40-c790-4639-8975-98e56da80ce9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f026f40-c790-4639-8975-98e56da80ce9-istiod-ca-cert\") pod \"3f026f40-c790-4639-8975-98e56da80ce9\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " Oct 30 21:23:07.847203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.847191 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-workload-socket\") pod \"3f026f40-c790-4639-8975-98e56da80ce9\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " Oct 30 21:23:07.847429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.847280 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f026f40-c790-4639-8975-98e56da80ce9-istio-token\" (UniqueName: \"kubernetes.io/projected/3f026f40-c790-4639-8975-98e56da80ce9-istio-token\") pod \"3f026f40-c790-4639-8975-98e56da80ce9\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " Oct 30 21:23:07.847429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.847387 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-credential-socket\") pod \"b42d62ed-b880-4a25-978c-47c703a5cdd3\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " Oct 30 21:23:07.847531 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.847432 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-credential-socket\") pod \"3f026f40-c790-4639-8975-98e56da80ce9\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " Oct 30 21:23:07.847531 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.847469 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f026f40-c790-4639-8975-98e56da80ce9-kube-api-access-6924b\" (UniqueName: \"kubernetes.io/projected/3f026f40-c790-4639-8975-98e56da80ce9-kube-api-access-6924b\") pod \"3f026f40-c790-4639-8975-98e56da80ce9\" (UID: \"3f026f40-c790-4639-8975-98e56da80ce9\") " Oct 30 21:23:07.847634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.847561 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-data\") pod \"b42d62ed-b880-4a25-978c-47c703a5cdd3\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " Oct 30 21:23:07.847634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.847590 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-workload-socket\") pod \"b42d62ed-b880-4a25-978c-47c703a5cdd3\" (UID: \"b42d62ed-b880-4a25-978c-47c703a5cdd3\") " Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.848125 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-workload-socket" pod "b42d62ed-b880-4a25-978c-47c703a5cdd3" (UID: "b42d62ed-b880-4a25-978c-47c703a5cdd3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.848433 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-credential-socket" pod "b42d62ed-b880-4a25-978c-47c703a5cdd3" (UID: "b42d62ed-b880-4a25-978c-47c703a5cdd3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.848522 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.848553 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqbjn\" (UniqueName: \"kubernetes.io/projected/37bf8b31-c305-4605-b262-f7f0aa6d2644-kube-api-access-tqbjn\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.848570 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.848587 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37bf8b31-c305-4605-b262-f7f0aa6d2644-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.848600 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.848619 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.848621 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-credential-socket" pod "3f026f40-c790-4639-8975-98e56da80ce9" (UID: "3f026f40-c790-4639-8975-98e56da80ce9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.848631 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83749c42-ef6a-4755-acbf-2166a604a6e6-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.848645 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.848657 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.848669 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.848683 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.848696 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.848709 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.848721 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.848735 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.848749 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83749c42-ef6a-4755-acbf-2166a604a6e6-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.848761 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83749c42-ef6a-4755-acbf-2166a604a6e6-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.848773 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37bf8b31-c305-4605-b262-f7f0aa6d2644-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.848787 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgn7q\" (UniqueName: \"kubernetes.io/projected/83749c42-ef6a-4755-acbf-2166a604a6e6-kube-api-access-rgn7q\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.849224 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-workload-socket" pod "3f026f40-c790-4639-8975-98e56da80ce9" (UID: "3f026f40-c790-4639-8975-98e56da80ce9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.849285 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b42d62ed-b880-4a25-978c-47c703a5cdd3-istiod-ca-cert" pod "b42d62ed-b880-4a25-978c-47c703a5cdd3" (UID: "b42d62ed-b880-4a25-978c-47c703a5cdd3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:07.849799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.849547 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-workload-certs" pod "3f026f40-c790-4639-8975-98e56da80ce9" (UID: "3f026f40-c790-4639-8975-98e56da80ce9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:07.851763 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.851724 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-envoy" pod "b42d62ed-b880-4a25-978c-47c703a5cdd3" (UID: "b42d62ed-b880-4a25-978c-47c703a5cdd3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:07.852556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.852062 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-token" pod "b42d62ed-b880-4a25-978c-47c703a5cdd3" (UID: "b42d62ed-b880-4a25-978c-47c703a5cdd3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:07.852556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.852294 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-istio-data" pod "3f026f40-c790-4639-8975-98e56da80ce9" (UID: "3f026f40-c790-4639-8975-98e56da80ce9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:07.853011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.852988 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-workload-certs" pod "b42d62ed-b880-4a25-978c-47c703a5cdd3" (UID: "b42d62ed-b880-4a25-978c-47c703a5cdd3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:07.853216 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.853193 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-data" pod "b42d62ed-b880-4a25-978c-47c703a5cdd3" (UID: "b42d62ed-b880-4a25-978c-47c703a5cdd3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:07.853445 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.853421 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f026f40-c790-4639-8975-98e56da80ce9-kube-api-access-6924b" pod "3f026f40-c790-4639-8975-98e56da80ce9" (UID: "3f026f40-c790-4639-8975-98e56da80ce9"). InnerVolumeSpecName "kube-api-access-6924b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:07.853881 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.853780 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f026f40-c790-4639-8975-98e56da80ce9-istiod-ca-cert" pod "3f026f40-c790-4639-8975-98e56da80ce9" (UID: "3f026f40-c790-4639-8975-98e56da80ce9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:07.855186 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.853955 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f026f40-c790-4639-8975-98e56da80ce9-istio-token" pod "3f026f40-c790-4639-8975-98e56da80ce9" (UID: "3f026f40-c790-4639-8975-98e56da80ce9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:07.855186 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.854236 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-podinfo" pod "b42d62ed-b880-4a25-978c-47c703a5cdd3" (UID: "b42d62ed-b880-4a25-978c-47c703a5cdd3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:23:07.855186 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.854590 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f026f40-c790-4639-8975-98e56da80ce9-istio-podinfo" pod "3f026f40-c790-4639-8975-98e56da80ce9" (UID: "3f026f40-c790-4639-8975-98e56da80ce9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:23:07.864081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.856066 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-istio-envoy" pod "3f026f40-c790-4639-8975-98e56da80ce9" (UID: "3f026f40-c790-4639-8975-98e56da80ce9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:07.864081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.856251 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b42d62ed-b880-4a25-978c-47c703a5cdd3-kube-api-access-52pbj" pod "b42d62ed-b880-4a25-978c-47c703a5cdd3" (UID: "b42d62ed-b880-4a25-978c-47c703a5cdd3"). InnerVolumeSpecName "kube-api-access-52pbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:07.950322 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.950294 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.950322 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.950326 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52pbj\" (UniqueName: \"kubernetes.io/projected/b42d62ed-b880-4a25-978c-47c703a5cdd3-kube-api-access-52pbj\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.950496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.950339 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.950496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.950348 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.950496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.950355 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.950496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.950363 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.950496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.950370 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.950496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.950377 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b42d62ed-b880-4a25-978c-47c703a5cdd3-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.950496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.950385 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f026f40-c790-4639-8975-98e56da80ce9-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.950496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.950392 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.950496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.950399 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f026f40-c790-4639-8975-98e56da80ce9-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.950496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.950408 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.950496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.950416 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.950496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.950423 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6924b\" (UniqueName: \"kubernetes.io/projected/3f026f40-c790-4639-8975-98e56da80ce9-kube-api-access-6924b\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.950496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.950435 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b42d62ed-b880-4a25-978c-47c703a5cdd3-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.950496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.950445 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f026f40-c790-4639-8975-98e56da80ce9-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:07.950496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:07.950452 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f026f40-c790-4639-8975-98e56da80ce9-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:08.402209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.402175 4926 generic.go:356] "Generic (PLEG): container finished" podID="3f026f40-c790-4639-8975-98e56da80ce9" containerID="4c721600dc428d4b2935c3d0fb16dce12e3a367fe25151615922d49e6296720d" exitCode=0 Oct 30 21:23:08.402727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.402254 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" event={"ID":"3f026f40-c790-4639-8975-98e56da80ce9","Type":"ContainerDied","Data":"4c721600dc428d4b2935c3d0fb16dce12e3a367fe25151615922d49e6296720d"} Oct 30 21:23:08.402727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.402313 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" event={"ID":"3f026f40-c790-4639-8975-98e56da80ce9","Type":"ContainerDied","Data":"09a2cb783d2e9636b6f869ce8f09a69aca2b2dbcd39e43f2f4ea5df42fbf2698"} Oct 30 21:23:08.402727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.402343 4926 scope.go:122] "RemoveContainer" containerID="4c721600dc428d4b2935c3d0fb16dce12e3a367fe25151615922d49e6296720d" Oct 30 21:23:08.402727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.402462 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp" Oct 30 21:23:08.405480 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.405452 4926 generic.go:356] "Generic (PLEG): container finished" podID="8a80b31b-f4f7-47bc-abe7-27b0505919f5" containerID="7ef02ebb7fa0b590d8c6acd6c7ae552e30b1c1c8d62e7547cf1a0baf65de5de8" exitCode=0 Oct 30 21:23:08.405610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.405561 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" event={"ID":"8a80b31b-f4f7-47bc-abe7-27b0505919f5","Type":"ContainerDied","Data":"7ef02ebb7fa0b590d8c6acd6c7ae552e30b1c1c8d62e7547cf1a0baf65de5de8"} Oct 30 21:23:08.414921 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.414890 4926 generic.go:356] "Generic (PLEG): container finished" podID="a3061a9a-e0c1-46c9-914b-a5aba158ec6a" containerID="6f0b7410464cb92dda1adffffb7ccf1e766b0a966d29d90b04e90754fb97bb65" exitCode=0 Oct 30 21:23:08.415183 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.415024 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" event={"ID":"a3061a9a-e0c1-46c9-914b-a5aba158ec6a","Type":"ContainerDied","Data":"6f0b7410464cb92dda1adffffb7ccf1e766b0a966d29d90b04e90754fb97bb65"} Oct 30 21:23:08.417204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.417175 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" event={"ID":"37bf8b31-c305-4605-b262-f7f0aa6d2644","Type":"ContainerDied","Data":"9a5b86b38ba65061ff818afd65b982250def54a82398b7759ec40cb20efda610"} Oct 30 21:23:08.417296 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.417285 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj" Oct 30 21:23:08.418990 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.418966 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" Oct 30 21:23:08.419133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.418987 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7" event={"ID":"83749c42-ef6a-4755-acbf-2166a604a6e6","Type":"ContainerDied","Data":"15573d8f2bc0712408b9af2583d9025801c3d2b23b762d33b40fe5872b73e0b8"} Oct 30 21:23:08.420397 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.420318 4926 scope.go:122] "RemoveContainer" containerID="4c721600dc428d4b2935c3d0fb16dce12e3a367fe25151615922d49e6296720d" Oct 30 21:23:08.420770 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:23:08.420738 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c721600dc428d4b2935c3d0fb16dce12e3a367fe25151615922d49e6296720d\": container with ID starting with 4c721600dc428d4b2935c3d0fb16dce12e3a367fe25151615922d49e6296720d not found: ID does not exist" containerID="4c721600dc428d4b2935c3d0fb16dce12e3a367fe25151615922d49e6296720d" Oct 30 21:23:08.420867 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.420775 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c721600dc428d4b2935c3d0fb16dce12e3a367fe25151615922d49e6296720d"} err="failed to get container status \"4c721600dc428d4b2935c3d0fb16dce12e3a367fe25151615922d49e6296720d\": rpc error: code = NotFound desc = could not find container \"4c721600dc428d4b2935c3d0fb16dce12e3a367fe25151615922d49e6296720d\": container with ID starting with 4c721600dc428d4b2935c3d0fb16dce12e3a367fe25151615922d49e6296720d not found: ID does not exist" Oct 30 21:23:08.420867 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.420794 4926 scope.go:122] "RemoveContainer" containerID="7fe18c40a71b364d1f49be81f1cc870d034725b5a29b299bde30419e5e8af284" Oct 30 21:23:08.420988 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.420956 4926 generic.go:356] "Generic (PLEG): container finished" podID="b42d62ed-b880-4a25-978c-47c703a5cdd3" containerID="eb143e0dcc55a237f94e1192b7449d8f1b34735672f2e91f4a70249e5d6cae62" exitCode=0 Oct 30 21:23:08.421095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.420992 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" event={"ID":"b42d62ed-b880-4a25-978c-47c703a5cdd3","Type":"ContainerDied","Data":"eb143e0dcc55a237f94e1192b7449d8f1b34735672f2e91f4a70249e5d6cae62"} Oct 30 21:23:08.421095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.421018 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" event={"ID":"b42d62ed-b880-4a25-978c-47c703a5cdd3","Type":"ContainerDied","Data":"b2c6af419e6768ff6f95627b06e6776daf8c5526183ecaaa2d1eb4e73f6b3055"} Oct 30 21:23:08.421221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.421120 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64" Oct 30 21:23:08.437403 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.437378 4926 scope.go:122] "RemoveContainer" containerID="aeb066d5d6f84bcd64676d129486de4ca6d8acc4ce81f3031664e5414d943cde" Oct 30 21:23:08.457611 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.457517 4926 scope.go:122] "RemoveContainer" containerID="eb143e0dcc55a237f94e1192b7449d8f1b34735672f2e91f4a70249e5d6cae62" Oct 30 21:23:08.470234 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.470197 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp"] Oct 30 21:23:08.482846 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.482816 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pn5fp"] Oct 30 21:23:08.483634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.483610 4926 scope.go:122] "RemoveContainer" containerID="eb143e0dcc55a237f94e1192b7449d8f1b34735672f2e91f4a70249e5d6cae62" Oct 30 21:23:08.484112 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:23:08.483999 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb143e0dcc55a237f94e1192b7449d8f1b34735672f2e91f4a70249e5d6cae62\": container with ID starting with eb143e0dcc55a237f94e1192b7449d8f1b34735672f2e91f4a70249e5d6cae62 not found: ID does not exist" containerID="eb143e0dcc55a237f94e1192b7449d8f1b34735672f2e91f4a70249e5d6cae62" Oct 30 21:23:08.484272 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.484245 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb143e0dcc55a237f94e1192b7449d8f1b34735672f2e91f4a70249e5d6cae62"} err="failed to get container status \"eb143e0dcc55a237f94e1192b7449d8f1b34735672f2e91f4a70249e5d6cae62\": rpc error: code = NotFound desc = could not find container \"eb143e0dcc55a237f94e1192b7449d8f1b34735672f2e91f4a70249e5d6cae62\": container with ID starting with eb143e0dcc55a237f94e1192b7449d8f1b34735672f2e91f4a70249e5d6cae62 not found: ID does not exist" Oct 30 21:23:08.535881 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.535295 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64"] Oct 30 21:23:08.542130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.542104 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-jtp64"] Oct 30 21:23:08.585241 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.585204 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7"] Oct 30 21:23:08.585983 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.585960 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-6t6s7"] Oct 30 21:23:08.615262 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.615238 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj"] Oct 30 21:23:08.628076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.626836 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-b4gkj"] Oct 30 21:23:08.645907 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.645885 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:23:08.663640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.663615 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-workload-socket\") pod \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " Oct 30 21:23:08.663759 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.663676 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-podinfo\") pod \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " Oct 30 21:23:08.663851 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.663806 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-envoy\") pod \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " Oct 30 21:23:08.663909 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.663877 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-credential-socket\") pod \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " Oct 30 21:23:08.663972 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.663916 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-workload-socket" pod "8a80b31b-f4f7-47bc-abe7-27b0505919f5" (UID: "8a80b31b-f4f7-47bc-abe7-27b0505919f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:08.664025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.663978 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a80b31b-f4f7-47bc-abe7-27b0505919f5-kube-api-access-qdmkm\" (UniqueName: \"kubernetes.io/projected/8a80b31b-f4f7-47bc-abe7-27b0505919f5-kube-api-access-qdmkm\") pod \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " Oct 30 21:23:08.664025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.664013 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istiod-ca-cert\") pod \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " Oct 30 21:23:08.664214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.664091 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-data\") pod \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " Oct 30 21:23:08.664214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.664195 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-workload-certs\") pod \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " Oct 30 21:23:08.664326 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.664248 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-token\" (UniqueName: \"kubernetes.io/projected/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-token\") pod \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\" (UID: \"8a80b31b-f4f7-47bc-abe7-27b0505919f5\") " Oct 30 21:23:08.664804 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.664775 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-credential-socket" pod "8a80b31b-f4f7-47bc-abe7-27b0505919f5" (UID: "8a80b31b-f4f7-47bc-abe7-27b0505919f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:08.664962 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.664942 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:08.665087 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.664969 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:08.665375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.665348 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-workload-certs" pod "8a80b31b-f4f7-47bc-abe7-27b0505919f5" (UID: "8a80b31b-f4f7-47bc-abe7-27b0505919f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:08.665474 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.665412 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-data" pod "8a80b31b-f4f7-47bc-abe7-27b0505919f5" (UID: "8a80b31b-f4f7-47bc-abe7-27b0505919f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:08.665529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.665477 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istiod-ca-cert" pod "8a80b31b-f4f7-47bc-abe7-27b0505919f5" (UID: "8a80b31b-f4f7-47bc-abe7-27b0505919f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:08.683048 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.680471 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-envoy" pod "8a80b31b-f4f7-47bc-abe7-27b0505919f5" (UID: "8a80b31b-f4f7-47bc-abe7-27b0505919f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:08.686626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.686539 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-podinfo" pod "8a80b31b-f4f7-47bc-abe7-27b0505919f5" (UID: "8a80b31b-f4f7-47bc-abe7-27b0505919f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:23:08.686626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.686564 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a80b31b-f4f7-47bc-abe7-27b0505919f5-kube-api-access-qdmkm" pod "8a80b31b-f4f7-47bc-abe7-27b0505919f5" (UID: "8a80b31b-f4f7-47bc-abe7-27b0505919f5"). InnerVolumeSpecName "kube-api-access-qdmkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:08.687388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.687352 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-token" pod "8a80b31b-f4f7-47bc-abe7-27b0505919f5" (UID: "8a80b31b-f4f7-47bc-abe7-27b0505919f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:08.720926 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.720898 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:23:08.765452 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.765421 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-envoy\") pod \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " Oct 30 21:23:08.765620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.765491 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-data\") pod \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " Oct 30 21:23:08.765620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.765559 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-workload-certs\") pod \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " Oct 30 21:23:08.765620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.765600 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-credential-socket\") pod \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " Oct 30 21:23:08.765782 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.765683 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-kube-api-access-rrh52\" (UniqueName: \"kubernetes.io/projected/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-kube-api-access-rrh52\") pod \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " Oct 30 21:23:08.765782 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.765729 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-workload-socket\") pod \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " Oct 30 21:23:08.765885 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.765790 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istiod-ca-cert\") pod \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " Oct 30 21:23:08.765885 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.765811 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-data" pod "a3061a9a-e0c1-46c9-914b-a5aba158ec6a" (UID: "a3061a9a-e0c1-46c9-914b-a5aba158ec6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:08.765885 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.765828 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-token\" (UniqueName: \"kubernetes.io/projected/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-token\") pod \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " Oct 30 21:23:08.766092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.766001 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-podinfo\") pod \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\" (UID: \"a3061a9a-e0c1-46c9-914b-a5aba158ec6a\") " Oct 30 21:23:08.766204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.766179 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-workload-socket" pod "a3061a9a-e0c1-46c9-914b-a5aba158ec6a" (UID: "a3061a9a-e0c1-46c9-914b-a5aba158ec6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:08.766487 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.766465 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istiod-ca-cert" pod "a3061a9a-e0c1-46c9-914b-a5aba158ec6a" (UID: "a3061a9a-e0c1-46c9-914b-a5aba158ec6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:08.766778 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.766757 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:08.766845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.766782 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:08.766845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.766796 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:08.766845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.766808 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:08.766845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.766821 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:08.766845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.766834 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdmkm\" (UniqueName: \"kubernetes.io/projected/8a80b31b-f4f7-47bc-abe7-27b0505919f5-kube-api-access-qdmkm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:08.767135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.766849 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:08.767135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.766862 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:08.767135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.766874 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a80b31b-f4f7-47bc-abe7-27b0505919f5-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:08.767135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.766886 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a80b31b-f4f7-47bc-abe7-27b0505919f5-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:08.768335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.767193 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-workload-certs" pod "a3061a9a-e0c1-46c9-914b-a5aba158ec6a" (UID: "a3061a9a-e0c1-46c9-914b-a5aba158ec6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:08.768335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.767321 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-credential-socket" pod "a3061a9a-e0c1-46c9-914b-a5aba158ec6a" (UID: "a3061a9a-e0c1-46c9-914b-a5aba158ec6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:08.770202 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.769710 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-envoy" pod "a3061a9a-e0c1-46c9-914b-a5aba158ec6a" (UID: "a3061a9a-e0c1-46c9-914b-a5aba158ec6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:23:08.771748 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.771723 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-token" pod "a3061a9a-e0c1-46c9-914b-a5aba158ec6a" (UID: "a3061a9a-e0c1-46c9-914b-a5aba158ec6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:08.771860 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.771814 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-kube-api-access-rrh52" pod "a3061a9a-e0c1-46c9-914b-a5aba158ec6a" (UID: "a3061a9a-e0c1-46c9-914b-a5aba158ec6a"). InnerVolumeSpecName "kube-api-access-rrh52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:08.771970 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.771950 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-podinfo" pod "a3061a9a-e0c1-46c9-914b-a5aba158ec6a" (UID: "a3061a9a-e0c1-46c9-914b-a5aba158ec6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:23:08.868094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.868063 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:08.868094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.868089 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:08.868094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.868098 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:08.868313 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.868108 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrh52\" (UniqueName: \"kubernetes.io/projected/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-kube-api-access-rrh52\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:08.868313 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.868117 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:08.868313 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:08.868127 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3061a9a-e0c1-46c9-914b-a5aba158ec6a-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:09.017797 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:09.017717 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37bf8b31-c305-4605-b262-f7f0aa6d2644" path="/var/lib/kubelet/pods/37bf8b31-c305-4605-b262-f7f0aa6d2644/volumes" Oct 30 21:23:09.018275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:09.018261 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f026f40-c790-4639-8975-98e56da80ce9" path="/var/lib/kubelet/pods/3f026f40-c790-4639-8975-98e56da80ce9/volumes" Oct 30 21:23:09.018705 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:09.018692 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83749c42-ef6a-4755-acbf-2166a604a6e6" path="/var/lib/kubelet/pods/83749c42-ef6a-4755-acbf-2166a604a6e6/volumes" Oct 30 21:23:09.019516 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:09.019489 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b42d62ed-b880-4a25-978c-47c703a5cdd3" path="/var/lib/kubelet/pods/b42d62ed-b880-4a25-978c-47c703a5cdd3/volumes" Oct 30 21:23:09.430714 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:09.430683 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" Oct 30 21:23:09.430714 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:09.430691 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9" event={"ID":"a3061a9a-e0c1-46c9-914b-a5aba158ec6a","Type":"ContainerDied","Data":"1274ee38906004678694db8a766122bf44a4e6e33b277c07aa7ae47d50b01647"} Oct 30 21:23:09.431241 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:09.430758 4926 scope.go:122] "RemoveContainer" containerID="6f0b7410464cb92dda1adffffb7ccf1e766b0a966d29d90b04e90754fb97bb65" Oct 30 21:23:09.437581 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:09.437550 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" event={"ID":"8a80b31b-f4f7-47bc-abe7-27b0505919f5","Type":"ContainerDied","Data":"a951f94531a5e5d193ee4fac3148ee7adb65f65bb19866bbc60ca1ae618261e2"} Oct 30 21:23:09.437720 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:09.437588 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9" Oct 30 21:23:09.442909 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:09.442888 4926 scope.go:122] "RemoveContainer" containerID="7ef02ebb7fa0b590d8c6acd6c7ae552e30b1c1c8d62e7547cf1a0baf65de5de8" Oct 30 21:23:09.491905 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:09.491873 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9"] Oct 30 21:23:09.506561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:09.506517 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-r2qx9"] Oct 30 21:23:09.537481 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:09.537446 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9"] Oct 30 21:23:09.543727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:09.543688 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-dx6x9"] Oct 30 21:23:10.432549 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.432516 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m"] Oct 30 21:23:10.433527 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433500 4926 cpu_manager.go:414] "containerMap: removing container" podUID="b42d62ed-b880-4a25-978c-47c703a5cdd3" containerName="istio-proxy" Oct 30 21:23:10.433527 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433527 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b42d62ed-b880-4a25-978c-47c703a5cdd3" containerName="istio-proxy" Oct 30 21:23:10.433704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433566 4926 cpu_manager.go:414] "containerMap: removing container" podUID="83749c42-ef6a-4755-acbf-2166a604a6e6" containerName="istio-proxy" Oct 30 21:23:10.433704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433575 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83749c42-ef6a-4755-acbf-2166a604a6e6" containerName="istio-proxy" Oct 30 21:23:10.433704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433588 4926 cpu_manager.go:414] "containerMap: removing container" podUID="8a80b31b-f4f7-47bc-abe7-27b0505919f5" containerName="istio-proxy" Oct 30 21:23:10.433704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433596 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a80b31b-f4f7-47bc-abe7-27b0505919f5" containerName="istio-proxy" Oct 30 21:23:10.433704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433609 4926 cpu_manager.go:414] "containerMap: removing container" podUID="5397fa31-83d0-4500-94bb-16e1848efe2f" containerName="istio-proxy" Oct 30 21:23:10.433704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433617 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5397fa31-83d0-4500-94bb-16e1848efe2f" containerName="istio-proxy" Oct 30 21:23:10.433704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433648 4926 cpu_manager.go:414] "containerMap: removing container" podUID="a3061a9a-e0c1-46c9-914b-a5aba158ec6a" containerName="istio-proxy" Oct 30 21:23:10.433704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433655 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3061a9a-e0c1-46c9-914b-a5aba158ec6a" containerName="istio-proxy" Oct 30 21:23:10.433704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433667 4926 cpu_manager.go:414] "containerMap: removing container" podUID="3f026f40-c790-4639-8975-98e56da80ce9" containerName="istio-proxy" Oct 30 21:23:10.433704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433674 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f026f40-c790-4639-8975-98e56da80ce9" containerName="istio-proxy" Oct 30 21:23:10.433704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433698 4926 cpu_manager.go:414] "containerMap: removing container" podUID="37bf8b31-c305-4605-b262-f7f0aa6d2644" containerName="istio-proxy" Oct 30 21:23:10.433704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433706 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37bf8b31-c305-4605-b262-f7f0aa6d2644" containerName="istio-proxy" Oct 30 21:23:10.434075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433724 4926 cpu_manager.go:414] "containerMap: removing container" podUID="f8da07db-83fb-49ad-950c-9206d47c0bec" containerName="revoke-keys" Oct 30 21:23:10.434075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433731 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8da07db-83fb-49ad-950c-9206d47c0bec" containerName="revoke-keys" Oct 30 21:23:10.434075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433886 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3061a9a-e0c1-46c9-914b-a5aba158ec6a" containerName="istio-proxy" Oct 30 21:23:10.434075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433898 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f026f40-c790-4639-8975-98e56da80ce9" containerName="istio-proxy" Oct 30 21:23:10.434075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433910 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="37bf8b31-c305-4605-b262-f7f0aa6d2644" containerName="istio-proxy" Oct 30 21:23:10.434075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433924 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8da07db-83fb-49ad-950c-9206d47c0bec" containerName="revoke-keys" Oct 30 21:23:10.434075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433938 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="b42d62ed-b880-4a25-978c-47c703a5cdd3" containerName="istio-proxy" Oct 30 21:23:10.434075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433950 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="83749c42-ef6a-4755-acbf-2166a604a6e6" containerName="istio-proxy" Oct 30 21:23:10.434075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433962 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a80b31b-f4f7-47bc-abe7-27b0505919f5" containerName="istio-proxy" Oct 30 21:23:10.434075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.433972 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="5397fa31-83d0-4500-94bb-16e1848efe2f" containerName="istio-proxy" Oct 30 21:23:10.438300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.438275 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.462629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.462006 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m"] Oct 30 21:23:10.492830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.492790 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-credential-socket\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.492995 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.492857 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr74r\" (UniqueName: \"kubernetes.io/projected/3eddcba6-e6e3-49cd-9078-5777d0faff13-kube-api-access-wr74r\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.492995 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.492912 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-data\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.493150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.493073 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-workload-socket\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.493150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.493102 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-workload-certs\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.493275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.493183 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-token\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.493275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.493203 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eddcba6-e6e3-49cd-9078-5777d0faff13-istiod-ca-cert\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.493275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.493246 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-podinfo\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.493275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.493274 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-envoy\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.597841 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.597685 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-podinfo\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.597841 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.597745 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-envoy\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.597841 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.597848 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-credential-socket\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.598149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.597903 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr74r\" (UniqueName: \"kubernetes.io/projected/3eddcba6-e6e3-49cd-9078-5777d0faff13-kube-api-access-wr74r\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.598149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.598097 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-data\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.602100 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.598269 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-workload-socket\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.602100 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.598436 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-workload-certs\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.602100 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.598571 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-token\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.602100 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.598598 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eddcba6-e6e3-49cd-9078-5777d0faff13-istiod-ca-cert\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.602100 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.600658 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eddcba6-e6e3-49cd-9078-5777d0faff13-istiod-ca-cert\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.602100 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.601379 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-workload-certs\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.602100 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.601601 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-workload-socket\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.602594 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.602184 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-credential-socket\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.603043 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.603003 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-data\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.607941 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.607918 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-envoy\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.608317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.608293 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-podinfo\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.633054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.625847 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr74r\" (UniqueName: \"kubernetes.io/projected/3eddcba6-e6e3-49cd-9078-5777d0faff13-kube-api-access-wr74r\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.633054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.628017 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-token\") pod \"e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:10.762787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:10.762697 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:11.017964 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:11.017890 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a80b31b-f4f7-47bc-abe7-27b0505919f5" path="/var/lib/kubelet/pods/8a80b31b-f4f7-47bc-abe7-27b0505919f5/volumes" Oct 30 21:23:11.018419 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:11.018404 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3061a9a-e0c1-46c9-914b-a5aba158ec6a" path="/var/lib/kubelet/pods/a3061a9a-e0c1-46c9-914b-a5aba158ec6a/volumes" Oct 30 21:23:11.089186 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:11.089152 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m"] Oct 30 21:23:11.090699 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:23:11.090672 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eddcba6_e6e3_49cd_9078_5777d0faff13.slice/crio-622fc89a8e3bd1b1bd1fa7191b5d8b763ce5c5ef1cd5dc5dc6441a5aa59edf52 WatchSource:0}: Error finding container 622fc89a8e3bd1b1bd1fa7191b5d8b763ce5c5ef1cd5dc5dc6441a5aa59edf52: Status 404 returned error can't find the container with id 622fc89a8e3bd1b1bd1fa7191b5d8b763ce5c5ef1cd5dc5dc6441a5aa59edf52 Oct 30 21:23:11.093906 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:11.093870 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:23:11.093983 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:11.093952 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:23:11.094028 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:11.093990 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:23:11.483360 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:11.483309 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" event={"ID":"3eddcba6-e6e3-49cd-9078-5777d0faff13","Type":"ContainerStarted","Data":"3ae73349604328c4e8f373d94b34f924fbedd2629a28e48e74e0d0c53683a4b4"} Oct 30 21:23:11.483823 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:11.483414 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" event={"ID":"3eddcba6-e6e3-49cd-9078-5777d0faff13","Type":"ContainerStarted","Data":"622fc89a8e3bd1b1bd1fa7191b5d8b763ce5c5ef1cd5dc5dc6441a5aa59edf52"} Oct 30 21:23:11.555664 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:11.555613 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" podStartSLOduration=1.555601367 podStartE2EDuration="1.555601367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:11.540978441 +0000 UTC m=+1378.662052194" watchObservedRunningTime="2026-10-30 21:23:11.555601367 +0000 UTC m=+1378.676675112" Oct 30 21:23:11.763843 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:11.763763 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:12.764236 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:12.764198 4926 patch_prober.go:29] interesting pod/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:23:12.764693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:12.764261 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" podUID="3eddcba6-e6e3-49cd-9078-5777d0faff13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 21:23:13.763174 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:13.763142 4926 patch_prober.go:29] interesting pod/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:23:13.763330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:13.763193 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" podUID="3eddcba6-e6e3-49cd-9078-5777d0faff13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 21:23:14.763241 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:14.763206 4926 patch_prober.go:29] interesting pod/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:23:14.763659 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:14.763265 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" podUID="3eddcba6-e6e3-49cd-9078-5777d0faff13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 21:23:14.825203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:14.825168 4926 scope.go:122] "RemoveContainer" containerID="fdbf942d59abb497bdfd50563a99825deb465879cfc45bb3461926743132aa9d" Oct 30 21:23:14.836511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:14.836488 4926 scope.go:122] "RemoveContainer" containerID="18023aa7e318917871c42f51f704d063dc98f022c262fbb691044d8ea36cebc2" Oct 30 21:23:14.845863 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:14.845844 4926 scope.go:122] "RemoveContainer" containerID="9b435b6e308da961b2acf352830265529a0cbbeeb043ec93a728dcafb8903d01" Oct 30 21:23:14.856453 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:14.856436 4926 scope.go:122] "RemoveContainer" containerID="c4f80056f277ad1febbdf8f8a9022de41b043217353d1f8cc65f1ed3d7794d6d" Oct 30 21:23:14.865941 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:14.865920 4926 scope.go:122] "RemoveContainer" containerID="3edef04b1358351b220e330f724d89ad620b4c28f6819e53eb7c67415dbb45ec" Oct 30 21:23:15.763790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:15.763753 4926 patch_prober.go:29] interesting pod/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:23:15.764308 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:15.763826 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" podUID="3eddcba6-e6e3-49cd-9078-5777d0faff13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 21:23:16.763782 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:16.763739 4926 patch_prober.go:29] interesting pod/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Oct 30 21:23:16.764000 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:16.763808 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" podUID="3eddcba6-e6e3-49cd-9078-5777d0faff13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Oct 30 21:23:17.769638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:17.769599 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:17.770151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:17.770063 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:17.770815 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:17.770795 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:23:24.334167 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:24.334128 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l"] Oct 30 21:23:24.340714 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:24.340690 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l" Oct 30 21:23:24.355940 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:24.355912 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l"] Oct 30 21:23:24.362123 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:24.362098 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:23:24.362278 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:24.362187 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:23:24.376694 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:24.376656 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxj7h\" (UniqueName: \"kubernetes.io/projected/ffdc19ca-36bf-48b8-a0ac-07cf550816dd-kube-api-access-pxj7h\") pod \"maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l\" (UID: \"ffdc19ca-36bf-48b8-a0ac-07cf550816dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l" Oct 30 21:23:24.376990 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:24.376969 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffdc19ca-36bf-48b8-a0ac-07cf550816dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l\" (UID: \"ffdc19ca-36bf-48b8-a0ac-07cf550816dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l" Oct 30 21:23:24.377151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:24.377133 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffdc19ca-36bf-48b8-a0ac-07cf550816dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l\" (UID: \"ffdc19ca-36bf-48b8-a0ac-07cf550816dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l" Oct 30 21:23:24.478734 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:24.478693 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxj7h\" (UniqueName: \"kubernetes.io/projected/ffdc19ca-36bf-48b8-a0ac-07cf550816dd-kube-api-access-pxj7h\") pod \"maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l\" (UID: \"ffdc19ca-36bf-48b8-a0ac-07cf550816dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l" Oct 30 21:23:24.478934 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:24.478772 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffdc19ca-36bf-48b8-a0ac-07cf550816dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l\" (UID: \"ffdc19ca-36bf-48b8-a0ac-07cf550816dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l" Oct 30 21:23:24.478934 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:24.478796 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffdc19ca-36bf-48b8-a0ac-07cf550816dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l\" (UID: \"ffdc19ca-36bf-48b8-a0ac-07cf550816dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l" Oct 30 21:23:24.479699 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:24.479674 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffdc19ca-36bf-48b8-a0ac-07cf550816dd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l\" (UID: \"ffdc19ca-36bf-48b8-a0ac-07cf550816dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l" Oct 30 21:23:24.481692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:24.481664 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffdc19ca-36bf-48b8-a0ac-07cf550816dd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l\" (UID: \"ffdc19ca-36bf-48b8-a0ac-07cf550816dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l" Oct 30 21:23:24.498024 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:24.497995 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxj7h\" (UniqueName: \"kubernetes.io/projected/ffdc19ca-36bf-48b8-a0ac-07cf550816dd-kube-api-access-pxj7h\") pod \"maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l\" (UID: \"ffdc19ca-36bf-48b8-a0ac-07cf550816dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l" Oct 30 21:23:24.655783 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:24.655700 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l" Oct 30 21:23:24.964299 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:24.964270 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l"] Oct 30 21:23:24.965562 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:23:24.965538 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffdc19ca_36bf_48b8_a0ac_07cf550816dd.slice/crio-5080737be3c5bfa861e91faebd30687ff4326bca3b304bd5440e11c58bd5a8c1 WatchSource:0}: Error finding container 5080737be3c5bfa861e91faebd30687ff4326bca3b304bd5440e11c58bd5a8c1: Status 404 returned error can't find the container with id 5080737be3c5bfa861e91faebd30687ff4326bca3b304bd5440e11c58bd5a8c1 Oct 30 21:23:25.688833 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:25.688795 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l" event={"ID":"ffdc19ca-36bf-48b8-a0ac-07cf550816dd","Type":"ContainerStarted","Data":"30bf9242bd8515e99f83c989685a860ab754da1238ed88ed05f6c9c43d1d4994"} Oct 30 21:23:25.689376 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:25.688864 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l" event={"ID":"ffdc19ca-36bf-48b8-a0ac-07cf550816dd","Type":"ContainerStarted","Data":"5080737be3c5bfa861e91faebd30687ff4326bca3b304bd5440e11c58bd5a8c1"} Oct 30 21:23:25.727073 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:25.727008 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l" podStartSLOduration=1.7269936129999999 podStartE2EDuration="1.726993613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:25.724981407 +0000 UTC m=+1392.846055153" watchObservedRunningTime="2026-10-30 21:23:25.726993613 +0000 UTC m=+1392.848067350" Oct 30 21:23:26.720055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:26.719232 4926 generic.go:356] "Generic (PLEG): container finished" podID="ffdc19ca-36bf-48b8-a0ac-07cf550816dd" containerID="30bf9242bd8515e99f83c989685a860ab754da1238ed88ed05f6c9c43d1d4994" exitCode=0 Oct 30 21:23:26.720055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:26.719311 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l" event={"ID":"ffdc19ca-36bf-48b8-a0ac-07cf550816dd","Type":"ContainerDied","Data":"30bf9242bd8515e99f83c989685a860ab754da1238ed88ed05f6c9c43d1d4994"} Oct 30 21:23:27.925265 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:27.925239 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l" Oct 30 21:23:28.035897 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:28.035809 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffdc19ca-36bf-48b8-a0ac-07cf550816dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffdc19ca-36bf-48b8-a0ac-07cf550816dd-maas-api-xks-ca\") pod \"ffdc19ca-36bf-48b8-a0ac-07cf550816dd\" (UID: \"ffdc19ca-36bf-48b8-a0ac-07cf550816dd\") " Oct 30 21:23:28.035897 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:28.035853 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffdc19ca-36bf-48b8-a0ac-07cf550816dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffdc19ca-36bf-48b8-a0ac-07cf550816dd-maas-api-openshift-ca\") pod \"ffdc19ca-36bf-48b8-a0ac-07cf550816dd\" (UID: \"ffdc19ca-36bf-48b8-a0ac-07cf550816dd\") " Oct 30 21:23:28.036191 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:28.035943 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffdc19ca-36bf-48b8-a0ac-07cf550816dd-kube-api-access-pxj7h\" (UniqueName: \"kubernetes.io/projected/ffdc19ca-36bf-48b8-a0ac-07cf550816dd-kube-api-access-pxj7h\") pod \"ffdc19ca-36bf-48b8-a0ac-07cf550816dd\" (UID: \"ffdc19ca-36bf-48b8-a0ac-07cf550816dd\") " Oct 30 21:23:28.036294 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:28.036263 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffdc19ca-36bf-48b8-a0ac-07cf550816dd-maas-api-openshift-ca" pod "ffdc19ca-36bf-48b8-a0ac-07cf550816dd" (UID: "ffdc19ca-36bf-48b8-a0ac-07cf550816dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:28.036634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:28.036614 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffdc19ca-36bf-48b8-a0ac-07cf550816dd-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:28.038072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:28.038028 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffdc19ca-36bf-48b8-a0ac-07cf550816dd-kube-api-access-pxj7h" pod "ffdc19ca-36bf-48b8-a0ac-07cf550816dd" (UID: "ffdc19ca-36bf-48b8-a0ac-07cf550816dd"). InnerVolumeSpecName "kube-api-access-pxj7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:28.038152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:28.038088 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffdc19ca-36bf-48b8-a0ac-07cf550816dd-maas-api-xks-ca" pod "ffdc19ca-36bf-48b8-a0ac-07cf550816dd" (UID: "ffdc19ca-36bf-48b8-a0ac-07cf550816dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:23:28.138317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:28.138285 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffdc19ca-36bf-48b8-a0ac-07cf550816dd-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:28.138317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:28.138316 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxj7h\" (UniqueName: \"kubernetes.io/projected/ffdc19ca-36bf-48b8-a0ac-07cf550816dd-kube-api-access-pxj7h\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:28.735321 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:28.735285 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l" Oct 30 21:23:28.735520 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:28.735339 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l" event={"ID":"ffdc19ca-36bf-48b8-a0ac-07cf550816dd","Type":"ContainerDied","Data":"5080737be3c5bfa861e91faebd30687ff4326bca3b304bd5440e11c58bd5a8c1"} Oct 30 21:23:28.735520 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:28.735385 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5080737be3c5bfa861e91faebd30687ff4326bca3b304bd5440e11c58bd5a8c1" Oct 30 21:23:30.955062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:30.953588 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz"] Oct 30 21:23:30.987085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:30.965524 4926 cpu_manager.go:414] "containerMap: removing container" podUID="ffdc19ca-36bf-48b8-a0ac-07cf550816dd" containerName="revoke-keys" Oct 30 21:23:30.987085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:30.965558 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffdc19ca-36bf-48b8-a0ac-07cf550816dd" containerName="revoke-keys" Oct 30 21:23:30.987085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:30.966547 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffdc19ca-36bf-48b8-a0ac-07cf550816dd" containerName="revoke-keys" Oct 30 21:23:31.058785 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:31.058746 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz" Oct 30 21:23:31.111552 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:31.111486 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:23:31.128430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:31.128396 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:23:31.249987 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:31.230119 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz\" (UID: \"ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz" Oct 30 21:23:31.249987 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:31.230191 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d4s2\" (UniqueName: \"kubernetes.io/projected/ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9-kube-api-access-9d4s2\") pod \"maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz\" (UID: \"ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz" Oct 30 21:23:31.249987 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:31.230268 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz\" (UID: \"ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz" Oct 30 21:23:31.249987 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:31.236232 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz"] Oct 30 21:23:31.336353 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:31.336315 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz\" (UID: \"ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz" Oct 30 21:23:31.336550 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:31.336388 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d4s2\" (UniqueName: \"kubernetes.io/projected/ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9-kube-api-access-9d4s2\") pod \"maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz\" (UID: \"ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz" Oct 30 21:23:31.336550 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:31.336470 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz\" (UID: \"ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz" Oct 30 21:23:31.337530 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:31.337501 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz\" (UID: \"ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz" Oct 30 21:23:31.340313 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:31.340253 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz\" (UID: \"ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz" Oct 30 21:23:31.357499 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:31.357478 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d4s2\" (UniqueName: \"kubernetes.io/projected/ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9-kube-api-access-9d4s2\") pod \"maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz\" (UID: \"ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz" Oct 30 21:23:31.543093 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:31.542998 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-s82n5"] Oct 30 21:23:31.549816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:31.549784 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-s82n5"] Oct 30 21:23:31.564440 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:31.564404 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz" Oct 30 21:23:31.953439 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:31.953210 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz"] Oct 30 21:23:31.957723 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:23:31.957693 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce610f9f_4bb7_4aab_a4a6_d4afe125e5b9.slice/crio-d7ed53ff407468237f47b22b3cba5c7c59e90081839db0e605d08e6d286af1f0 WatchSource:0}: Error finding container d7ed53ff407468237f47b22b3cba5c7c59e90081839db0e605d08e6d286af1f0: Status 404 returned error can't find the container with id d7ed53ff407468237f47b22b3cba5c7c59e90081839db0e605d08e6d286af1f0 Oct 30 21:23:32.958786 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:32.958748 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz" event={"ID":"ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9","Type":"ContainerStarted","Data":"6652831f2eafc1ec18c19be3f78d64ce09e1e9f584a45dd4dc13905e8c57bc2a"} Oct 30 21:23:32.959226 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:32.958805 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz" event={"ID":"ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9","Type":"ContainerStarted","Data":"d7ed53ff407468237f47b22b3cba5c7c59e90081839db0e605d08e6d286af1f0"} Oct 30 21:23:32.996462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:32.996402 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz" podStartSLOduration=2.996384857 podStartE2EDuration="2.996384857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:32.994003198 +0000 UTC m=+1400.115076945" watchObservedRunningTime="2026-10-30 21:23:32.996384857 +0000 UTC m=+1400.117458605" Oct 30 21:23:33.020569 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:33.020536 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ebc545e-7736-47fb-af0d-ab6cfd748fad" path="/var/lib/kubelet/pods/8ebc545e-7736-47fb-af0d-ab6cfd748fad/volumes" Oct 30 21:23:33.969599 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:33.969561 4926 generic.go:356] "Generic (PLEG): container finished" podID="ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9" containerID="6652831f2eafc1ec18c19be3f78d64ce09e1e9f584a45dd4dc13905e8c57bc2a" exitCode=0 Oct 30 21:23:33.970103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:33.969656 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz" event={"ID":"ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9","Type":"ContainerDied","Data":"6652831f2eafc1ec18c19be3f78d64ce09e1e9f584a45dd4dc13905e8c57bc2a"} Oct 30 21:23:35.174265 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:35.174241 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz" Oct 30 21:23:35.207798 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:35.207756 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9-maas-api-openshift-ca\") pod \"ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9\" (UID: \"ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9\") " Oct 30 21:23:35.207991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:35.207850 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9-maas-api-xks-ca\") pod \"ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9\" (UID: \"ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9\") " Oct 30 21:23:35.208118 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:35.208101 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9-kube-api-access-9d4s2\" (UniqueName: \"kubernetes.io/projected/ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9-kube-api-access-9d4s2\") pod \"ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9\" (UID: \"ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9\") " Oct 30 21:23:35.208224 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:35.208195 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9-maas-api-openshift-ca" pod "ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9" (UID: "ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:23:35.208912 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:35.208883 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:35.210407 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:35.210373 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9-kube-api-access-9d4s2" pod "ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9" (UID: "ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9"). InnerVolumeSpecName "kube-api-access-9d4s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:23:35.210565 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:35.210451 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9-maas-api-xks-ca" pod "ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9" (UID: "ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:23:35.310605 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:35.310491 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:35.310605 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:35.310529 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d4s2\" (UniqueName: \"kubernetes.io/projected/ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9-kube-api-access-9d4s2\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:23:36.011058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:36.006491 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz" event={"ID":"ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9","Type":"ContainerDied","Data":"d7ed53ff407468237f47b22b3cba5c7c59e90081839db0e605d08e6d286af1f0"} Oct 30 21:23:36.011058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:36.006571 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7ed53ff407468237f47b22b3cba5c7c59e90081839db0e605d08e6d286af1f0" Oct 30 21:23:36.011058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:36.006657 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz" Oct 30 21:23:37.489139 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:23:37.489092 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:23:48.765160 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.765124 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z"] Oct 30 21:23:48.765851 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.765833 4926 cpu_manager.go:414] "containerMap: removing container" podUID="ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9" containerName="revoke-keys" Oct 30 21:23:48.765899 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.765853 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9" containerName="revoke-keys" Oct 30 21:23:48.766019 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.766009 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9" containerName="revoke-keys" Oct 30 21:23:48.771027 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.771007 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.808093 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.808015 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-ca18b3-openshift-default-dockercfg-kl4ft\"" Oct 30 21:23:48.808268 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.808231 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z"] Oct 30 21:23:48.863970 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.863928 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-workload-certs\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.864179 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.864085 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/226ee133-aef2-4cd2-b0bb-5bb745a03862-istiod-ca-cert\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.864179 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.864122 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-workload-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.864317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.864198 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-podinfo\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.864372 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.864335 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-data\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.864429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.864370 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-envoy\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.864429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.864406 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-credential-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.864511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.864494 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-token\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.864558 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.864520 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbfc5\" (UniqueName: \"kubernetes.io/projected/226ee133-aef2-4cd2-b0bb-5bb745a03862-kube-api-access-nbfc5\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.965853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.965818 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-workload-certs\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.966063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.965899 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/226ee133-aef2-4cd2-b0bb-5bb745a03862-istiod-ca-cert\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.966063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.965920 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-workload-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.966063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.965948 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-podinfo\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.966063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.966058 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-data\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.966340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.966090 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-envoy\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.966340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.966120 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-credential-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.966340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.966183 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-token\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.966340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.966304 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbfc5\" (UniqueName: \"kubernetes.io/projected/226ee133-aef2-4cd2-b0bb-5bb745a03862-kube-api-access-nbfc5\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.966555 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.966372 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-workload-certs\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.966719 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.966694 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-workload-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.966954 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.966931 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-credential-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.967122 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.967083 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-data\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.967122 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.966953 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/226ee133-aef2-4cd2-b0bb-5bb745a03862-istiod-ca-cert\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.968859 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.968834 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-envoy\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.969271 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.969238 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-podinfo\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.992094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.992026 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-token\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:48.992236 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:48.992108 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbfc5\" (UniqueName: \"kubernetes.io/projected/226ee133-aef2-4cd2-b0bb-5bb745a03862-kube-api-access-nbfc5\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:49.114323 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:49.114230 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:49.446919 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:23:49.446883 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226ee133_aef2_4cd2_b0bb_5bb745a03862.slice/crio-33dbea3bb03ae53123c1b7db31babc5399ab5926f56e8fba7e0d16726469660b WatchSource:0}: Error finding container 33dbea3bb03ae53123c1b7db31babc5399ab5926f56e8fba7e0d16726469660b: Status 404 returned error can't find the container with id 33dbea3bb03ae53123c1b7db31babc5399ab5926f56e8fba7e0d16726469660b Oct 30 21:23:49.448266 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:49.448229 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z"] Oct 30 21:23:49.450323 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:49.450284 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:23:49.450425 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:49.450350 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:23:49.450425 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:49.450378 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:23:50.146118 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:50.146078 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" event={"ID":"226ee133-aef2-4cd2-b0bb-5bb745a03862","Type":"ContainerStarted","Data":"05cfd4fb21377a75dd0b48285493154c297d94a6421f5f7f84c4dd3411b75e14"} Oct 30 21:23:50.146495 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:50.146142 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" event={"ID":"226ee133-aef2-4cd2-b0bb-5bb745a03862","Type":"ContainerStarted","Data":"33dbea3bb03ae53123c1b7db31babc5399ab5926f56e8fba7e0d16726469660b"} Oct 30 21:23:50.217589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:50.217534 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" podStartSLOduration=2.217516947 podStartE2EDuration="2.217516947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:50.212072855 +0000 UTC m=+1417.333146604" watchObservedRunningTime="2026-10-30 21:23:50.217516947 +0000 UTC m=+1417.338590694" Oct 30 21:23:51.114947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:51.114916 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:51.116497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:51.116469 4926 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:23:51.116639 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:51.116519 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" podUID="226ee133-aef2-4cd2-b0bb-5bb745a03862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 21:23:52.121065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:52.120455 4926 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:23:52.121065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:52.120518 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" podUID="226ee133-aef2-4cd2-b0bb-5bb745a03862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 21:23:53.115261 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:53.115224 4926 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:23:53.115477 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:53.115279 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" podUID="226ee133-aef2-4cd2-b0bb-5bb745a03862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 21:23:54.115129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:54.115082 4926 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:23:54.115540 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:54.115150 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" podUID="226ee133-aef2-4cd2-b0bb-5bb745a03862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 21:23:55.115081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:55.115023 4926 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:23:55.115316 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:55.115108 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" podUID="226ee133-aef2-4cd2-b0bb-5bb745a03862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 21:23:56.142511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.142456 4926 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 30 21:23:56.151353 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.151308 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" podUID="226ee133-aef2-4cd2-b0bb-5bb745a03862" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 30 21:23:56.296058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.292839 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv"] Oct 30 21:23:56.560120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.512471 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.560120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.534594 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-c1da2183-openshift-default-dockercfg-gj6jg\"" Oct 30 21:23:56.646123 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.646081 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv"] Oct 30 21:23:56.646404 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.646389 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-057c1dfd-pqhtt"] Oct 30 21:23:56.670071 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.653434 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-workload-socket\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.670071 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.653488 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce21589-5994-4cd8-b978-450027ab13ec-istiod-ca-cert\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.670071 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.653537 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce21589-5994-4cd8-b978-450027ab13ec-istio-podinfo\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.670071 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.657096 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-credential-socket\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.670071 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.657161 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-istio-data\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.670071 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.657189 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5rgl\" (UniqueName: \"kubernetes.io/projected/1ce21589-5994-4cd8-b978-450027ab13ec-kube-api-access-n5rgl\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.670071 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.657280 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce21589-5994-4cd8-b978-450027ab13ec-istio-token\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.670071 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.657326 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-workload-certs\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.670071 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.657609 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-istio-envoy\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.716382 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.716348 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-057c1dfd-pqhtt" Oct 30 21:23:56.735171 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.735135 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-057c1dfd-pqhtt"] Oct 30 21:23:56.743904 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.743767 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:23:56.744277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.744257 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:23:56.758622 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.758595 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-credential-socket\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.758795 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.758637 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-istio-data\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.758876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.758788 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5rgl\" (UniqueName: \"kubernetes.io/projected/1ce21589-5994-4cd8-b978-450027ab13ec-kube-api-access-n5rgl\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.758936 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.758874 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0927fd-aba0-4ab2-8e77-85c03e39dcbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-057c1dfd-pqhtt\" (UID: \"ba0927fd-aba0-4ab2-8e77-85c03e39dcbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-057c1dfd-pqhtt" Oct 30 21:23:56.758936 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.758927 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce21589-5994-4cd8-b978-450027ab13ec-istio-token\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.759119 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.758975 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-workload-certs\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.759119 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.759078 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-credential-socket\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.759314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.759291 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf2zh\" (UniqueName: \"kubernetes.io/projected/ba0927fd-aba0-4ab2-8e77-85c03e39dcbd-kube-api-access-vf2zh\") pod \"maas-api-revoke-sub-057c1dfd-pqhtt\" (UID: \"ba0927fd-aba0-4ab2-8e77-85c03e39dcbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-057c1dfd-pqhtt" Oct 30 21:23:56.759432 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.759413 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-istio-envoy\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.759488 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.759470 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-workload-certs\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.759638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.759512 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-workload-socket\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.759683 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.759651 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce21589-5994-4cd8-b978-450027ab13ec-istiod-ca-cert\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.759724 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.759689 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce21589-5994-4cd8-b978-450027ab13ec-istio-podinfo\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.759760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.759727 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0927fd-aba0-4ab2-8e77-85c03e39dcbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-057c1dfd-pqhtt\" (UID: \"ba0927fd-aba0-4ab2-8e77-85c03e39dcbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-057c1dfd-pqhtt" Oct 30 21:23:56.759760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.759742 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-workload-socket\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.759834 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.759569 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-istio-data\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.760291 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.760274 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce21589-5994-4cd8-b978-450027ab13ec-istiod-ca-cert\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.762507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.762483 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce21589-5994-4cd8-b978-450027ab13ec-istio-podinfo\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.762764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.762746 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-istio-envoy\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.819107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.818920 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5rgl\" (UniqueName: \"kubernetes.io/projected/1ce21589-5994-4cd8-b978-450027ab13ec-kube-api-access-n5rgl\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.832766 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.831665 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce21589-5994-4cd8-b978-450027ab13ec-istio-token\") pod \"e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:56.869054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.865092 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0927fd-aba0-4ab2-8e77-85c03e39dcbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-057c1dfd-pqhtt\" (UID: \"ba0927fd-aba0-4ab2-8e77-85c03e39dcbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-057c1dfd-pqhtt" Oct 30 21:23:56.869054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.865304 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf2zh\" (UniqueName: \"kubernetes.io/projected/ba0927fd-aba0-4ab2-8e77-85c03e39dcbd-kube-api-access-vf2zh\") pod \"maas-api-revoke-sub-057c1dfd-pqhtt\" (UID: \"ba0927fd-aba0-4ab2-8e77-85c03e39dcbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-057c1dfd-pqhtt" Oct 30 21:23:56.869054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.865492 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0927fd-aba0-4ab2-8e77-85c03e39dcbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-057c1dfd-pqhtt\" (UID: \"ba0927fd-aba0-4ab2-8e77-85c03e39dcbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-057c1dfd-pqhtt" Oct 30 21:23:56.893054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.892484 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0927fd-aba0-4ab2-8e77-85c03e39dcbd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-057c1dfd-pqhtt\" (UID: \"ba0927fd-aba0-4ab2-8e77-85c03e39dcbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-057c1dfd-pqhtt" Oct 30 21:23:56.893054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.892809 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0927fd-aba0-4ab2-8e77-85c03e39dcbd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-057c1dfd-pqhtt\" (UID: \"ba0927fd-aba0-4ab2-8e77-85c03e39dcbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-057c1dfd-pqhtt" Oct 30 21:23:56.910533 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.910371 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf2zh\" (UniqueName: \"kubernetes.io/projected/ba0927fd-aba0-4ab2-8e77-85c03e39dcbd-kube-api-access-vf2zh\") pod \"maas-api-revoke-sub-057c1dfd-pqhtt\" (UID: \"ba0927fd-aba0-4ab2-8e77-85c03e39dcbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-057c1dfd-pqhtt" Oct 30 21:23:56.961679 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:56.961639 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:57.057051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:57.054479 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-057c1dfd-pqhtt" Oct 30 21:23:57.306658 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:57.306621 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv"] Oct 30 21:23:57.311650 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:23:57.311619 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce21589_5994_4cd8_b978_450027ab13ec.slice/crio-159acd8e35b97fb2f0a417e8e7ee434e57f9cd615f844e32f21808c4d4ee10af WatchSource:0}: Error finding container 159acd8e35b97fb2f0a417e8e7ee434e57f9cd615f844e32f21808c4d4ee10af: Status 404 returned error can't find the container with id 159acd8e35b97fb2f0a417e8e7ee434e57f9cd615f844e32f21808c4d4ee10af Oct 30 21:23:57.314678 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:57.314641 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:23:57.314786 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:57.314722 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:23:57.314786 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:57.314775 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:23:57.438366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:57.438274 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:57.440081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:57.439805 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:57.441062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:57.441028 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:23:57.454129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:57.454101 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-057c1dfd-pqhtt"] Oct 30 21:23:57.641178 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:57.641090 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-057c1dfd-pqhtt" event={"ID":"ba0927fd-aba0-4ab2-8e77-85c03e39dcbd","Type":"ContainerStarted","Data":"d19fad5a74d1933adac2e75438881c94495fbd15a2735d4a3737c5b30391d3e4"} Oct 30 21:23:57.644144 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:57.644108 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" event={"ID":"1ce21589-5994-4cd8-b978-450027ab13ec","Type":"ContainerStarted","Data":"53f2daf36737829dddb318b004c58df08401cfa4ba55c098c16dabdfa52ee220"} Oct 30 21:23:57.644317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:57.644172 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" event={"ID":"1ce21589-5994-4cd8-b978-450027ab13ec","Type":"ContainerStarted","Data":"159acd8e35b97fb2f0a417e8e7ee434e57f9cd615f844e32f21808c4d4ee10af"} Oct 30 21:23:57.689277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:57.689209 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" podStartSLOduration=1.689178187 podStartE2EDuration="1.689178187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:23:57.687656483 +0000 UTC m=+1424.808730233" watchObservedRunningTime="2026-10-30 21:23:57.689178187 +0000 UTC m=+1424.810251937" Oct 30 21:23:57.961937 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:57.961897 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:23:58.653272 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:58.653232 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-057c1dfd-pqhtt" event={"ID":"ba0927fd-aba0-4ab2-8e77-85c03e39dcbd","Type":"ContainerStarted","Data":"7820a46d321e6dbb0d95db8bb327673d6600de4e4a12c7752452a9cf16bd2c31"} Oct 30 21:23:58.964513 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:58.964483 4926 patch_prober.go:29] interesting pod/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:23:58.964687 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:58.964532 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" podUID="1ce21589-5994-4cd8-b978-450027ab13ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 21:23:59.664197 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:59.664162 4926 generic.go:356] "Generic (PLEG): container finished" podID="ba0927fd-aba0-4ab2-8e77-85c03e39dcbd" containerID="7820a46d321e6dbb0d95db8bb327673d6600de4e4a12c7752452a9cf16bd2c31" exitCode=0 Oct 30 21:23:59.664623 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:59.664250 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-057c1dfd-pqhtt" event={"ID":"ba0927fd-aba0-4ab2-8e77-85c03e39dcbd","Type":"ContainerDied","Data":"7820a46d321e6dbb0d95db8bb327673d6600de4e4a12c7752452a9cf16bd2c31"} Oct 30 21:23:59.962405 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:59.962330 4926 patch_prober.go:29] interesting pod/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:23:59.962405 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:23:59.962383 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" podUID="1ce21589-5994-4cd8-b978-450027ab13ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 21:24:00.952367 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:00.952305 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-057c1dfd-pqhtt" Oct 30 21:24:00.962239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:00.962210 4926 patch_prober.go:29] interesting pod/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:24:00.962354 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:00.962271 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" podUID="1ce21589-5994-4cd8-b978-450027ab13ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 21:24:01.060940 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:01.060901 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba0927fd-aba0-4ab2-8e77-85c03e39dcbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0927fd-aba0-4ab2-8e77-85c03e39dcbd-maas-api-xks-ca\") pod \"ba0927fd-aba0-4ab2-8e77-85c03e39dcbd\" (UID: \"ba0927fd-aba0-4ab2-8e77-85c03e39dcbd\") " Oct 30 21:24:01.061214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:01.061191 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba0927fd-aba0-4ab2-8e77-85c03e39dcbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0927fd-aba0-4ab2-8e77-85c03e39dcbd-maas-api-openshift-ca\") pod \"ba0927fd-aba0-4ab2-8e77-85c03e39dcbd\" (UID: \"ba0927fd-aba0-4ab2-8e77-85c03e39dcbd\") " Oct 30 21:24:01.061697 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:01.061669 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0927fd-aba0-4ab2-8e77-85c03e39dcbd-maas-api-openshift-ca" pod "ba0927fd-aba0-4ab2-8e77-85c03e39dcbd" (UID: "ba0927fd-aba0-4ab2-8e77-85c03e39dcbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:01.061901 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:01.061697 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0927fd-aba0-4ab2-8e77-85c03e39dcbd-kube-api-access-vf2zh\" (UniqueName: \"kubernetes.io/projected/ba0927fd-aba0-4ab2-8e77-85c03e39dcbd-kube-api-access-vf2zh\") pod \"ba0927fd-aba0-4ab2-8e77-85c03e39dcbd\" (UID: \"ba0927fd-aba0-4ab2-8e77-85c03e39dcbd\") " Oct 30 21:24:01.063214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:01.063189 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ba0927fd-aba0-4ab2-8e77-85c03e39dcbd-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:01.137980 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:01.137925 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba0927fd-aba0-4ab2-8e77-85c03e39dcbd-maas-api-xks-ca" pod "ba0927fd-aba0-4ab2-8e77-85c03e39dcbd" (UID: "ba0927fd-aba0-4ab2-8e77-85c03e39dcbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:01.139060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:01.139001 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0927fd-aba0-4ab2-8e77-85c03e39dcbd-kube-api-access-vf2zh" pod "ba0927fd-aba0-4ab2-8e77-85c03e39dcbd" (UID: "ba0927fd-aba0-4ab2-8e77-85c03e39dcbd"). InnerVolumeSpecName "kube-api-access-vf2zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:01.165732 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:01.165695 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ba0927fd-aba0-4ab2-8e77-85c03e39dcbd-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:01.165967 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:01.165947 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vf2zh\" (UniqueName: \"kubernetes.io/projected/ba0927fd-aba0-4ab2-8e77-85c03e39dcbd-kube-api-access-vf2zh\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:01.718806 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:01.718768 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-057c1dfd-pqhtt" Oct 30 21:24:01.718806 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:01.718788 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-057c1dfd-pqhtt" event={"ID":"ba0927fd-aba0-4ab2-8e77-85c03e39dcbd","Type":"ContainerDied","Data":"d19fad5a74d1933adac2e75438881c94495fbd15a2735d4a3737c5b30391d3e4"} Oct 30 21:24:01.719076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:01.718852 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d19fad5a74d1933adac2e75438881c94495fbd15a2735d4a3737c5b30391d3e4" Oct 30 21:24:01.963026 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:01.962982 4926 patch_prober.go:29] interesting pod/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:24:01.963583 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:01.963104 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" podUID="1ce21589-5994-4cd8-b978-450027ab13ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 21:24:02.961953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:02.961912 4926 patch_prober.go:29] interesting pod/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Oct 30 21:24:02.962149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:02.961971 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" podUID="1ce21589-5994-4cd8-b978-450027ab13ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Oct 30 21:24:03.985287 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:03.985251 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:24:03.985947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:03.985922 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:24:03.986679 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:03.986653 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:24:05.454870 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:05.454833 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:24:07.490402 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:24:07.490360 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:24:18.911053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:18.898777 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk"] Oct 30 21:24:18.911053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:18.901155 4926 cpu_manager.go:414] "containerMap: removing container" podUID="ba0927fd-aba0-4ab2-8e77-85c03e39dcbd" containerName="revoke-keys" Oct 30 21:24:18.911053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:18.901176 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0927fd-aba0-4ab2-8e77-85c03e39dcbd" containerName="revoke-keys" Oct 30 21:24:18.911053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:18.901912 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0927fd-aba0-4ab2-8e77-85c03e39dcbd" containerName="revoke-keys" Oct 30 21:24:18.914491 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:18.914456 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:18.928809 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:18.928775 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk"] Oct 30 21:24:19.041499 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.041467 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-envoy\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.041687 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.041628 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6eca6b-e95a-44c5-bc67-f431a266729d-istiod-ca-cert\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.041741 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.041725 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-workload-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.041780 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.041748 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-credential-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.041811 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.041784 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-data\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.041901 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.041884 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-token\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.041995 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.041978 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtvfd\" (UniqueName: \"kubernetes.io/projected/0e6eca6b-e95a-44c5-bc67-f431a266729d-kube-api-access-rtvfd\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.042028 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.042010 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-workload-certs\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.042116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.042075 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-podinfo\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.142990 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.142950 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6eca6b-e95a-44c5-bc67-f431a266729d-istiod-ca-cert\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.143210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.143056 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-workload-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.143210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.143072 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-credential-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.143309 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.143205 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-data\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.143309 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.143267 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-token\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.143764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.143504 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-workload-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.143764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.143550 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtvfd\" (UniqueName: \"kubernetes.io/projected/0e6eca6b-e95a-44c5-bc67-f431a266729d-kube-api-access-rtvfd\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.143764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.143581 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-credential-socket\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.143764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.143614 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-workload-certs\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.143764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.143644 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-podinfo\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.143764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.143688 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-data\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.143764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.143704 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-envoy\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.144233 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.143824 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6eca6b-e95a-44c5-bc67-f431a266729d-istiod-ca-cert\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.144233 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.144024 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-workload-certs\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.145930 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.145910 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-envoy\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.146189 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.146170 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-podinfo\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.162546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.162488 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-token\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.162638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.162566 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtvfd\" (UniqueName: \"kubernetes.io/projected/0e6eca6b-e95a-44c5-bc67-f431a266729d-kube-api-access-rtvfd\") pod \"e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.239829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.239786 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:19.591791 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:24:19.591745 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6eca6b_e95a_44c5_bc67_f431a266729d.slice/crio-314beda9e5a2d1625d292bf1a3c239810d3494cf51714696194d2a5ee4e4b800 WatchSource:0}: Error finding container 314beda9e5a2d1625d292bf1a3c239810d3494cf51714696194d2a5ee4e4b800: Status 404 returned error can't find the container with id 314beda9e5a2d1625d292bf1a3c239810d3494cf51714696194d2a5ee4e4b800 Oct 30 21:24:19.593515 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.593482 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk"] Oct 30 21:24:19.595431 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.595348 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:24:19.595549 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.595510 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:24:19.595612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.595557 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:24:19.954816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.954772 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" event={"ID":"0e6eca6b-e95a-44c5-bc67-f431a266729d","Type":"ContainerStarted","Data":"7f2ace4c3b3c01f884d885dee065419c64fe53d61c010c2fb2e8b40a4eff8cc7"} Oct 30 21:24:19.955358 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:19.954845 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" event={"ID":"0e6eca6b-e95a-44c5-bc67-f431a266729d","Type":"ContainerStarted","Data":"314beda9e5a2d1625d292bf1a3c239810d3494cf51714696194d2a5ee4e4b800"} Oct 30 21:24:20.240364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:20.240267 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:20.242892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:20.242840 4926 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:24:20.243519 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:20.242945 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" podUID="0e6eca6b-e95a-44c5-bc67-f431a266729d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:24:21.240678 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:21.240644 4926 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:24:21.241191 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:21.240706 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" podUID="0e6eca6b-e95a-44c5-bc67-f431a266729d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:24:22.241133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:22.241098 4926 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:24:22.241510 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:22.241159 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" podUID="0e6eca6b-e95a-44c5-bc67-f431a266729d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:24:23.240580 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:23.240539 4926 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:24:23.240964 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:23.240935 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" podUID="0e6eca6b-e95a-44c5-bc67-f431a266729d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:24:24.240225 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:24.240183 4926 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:24:24.240741 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:24.240248 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" podUID="0e6eca6b-e95a-44c5-bc67-f431a266729d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:24:25.240492 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:25.240459 4926 patch_prober.go:29] interesting pod/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body= Oct 30 21:24:25.240908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:25.240511 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" podUID="0e6eca6b-e95a-44c5-bc67-f431a266729d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" Oct 30 21:24:26.247150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:26.247118 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:26.247598 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:26.247464 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:26.248268 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:26.248244 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:24:26.300072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:26.300000 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" podStartSLOduration=8.299985498 podStartE2EDuration="8.299985498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:24:20.02009877 +0000 UTC m=+1447.141172518" watchObservedRunningTime="2026-10-30 21:24:26.299985498 +0000 UTC m=+1453.421059245" Oct 30 21:24:31.925162 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:31.925098 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l"] Oct 30 21:24:31.943113 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:31.942181 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-59fb17-zmf8l"] Oct 30 21:24:32.809522 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:32.809489 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz"] Oct 30 21:24:32.819976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:32.819947 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0e7510-7bmmz"] Oct 30 21:24:33.027252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:33.027218 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9" path="/var/lib/kubelet/pods/ce610f9f-4bb7-4aab-a4a6-d4afe125e5b9/volumes" Oct 30 21:24:33.028020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:33.027995 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffdc19ca-36bf-48b8-a0ac-07cf550816dd" path="/var/lib/kubelet/pods/ffdc19ca-36bf-48b8-a0ac-07cf550816dd/volumes" Oct 30 21:24:37.521519 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:24:37.521473 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:24:38.744093 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:38.741901 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx"] Oct 30 21:24:38.744093 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:38.742185 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" podUID="4aca75f2-2a47-484f-b19b-41c6d228cb2c" containerName="istio-proxy" containerID="cri-o://c2c8aa633a3df83d526b04a2ba33b53738de3747d6880c78ff7612d231275061" gracePeriod=30 Oct 30 21:24:38.748443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:38.748404 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m"] Oct 30 21:24:38.748827 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:38.748781 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" podUID="3eddcba6-e6e3-49cd-9078-5777d0faff13" containerName="istio-proxy" containerID="cri-o://3ae73349604328c4e8f373d94b34f924fbedd2629a28e48e74e0d0c53683a4b4" gracePeriod=30 Oct 30 21:24:39.354332 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:39.354300 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4266b7-p94mm"] Oct 30 21:24:39.358925 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:39.358908 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4266b7-p94mm" Oct 30 21:24:39.400642 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:39.400609 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4266b7-p94mm"] Oct 30 21:24:39.434648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:39.434617 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:24:39.434648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:39.434646 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:24:39.448870 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:39.448842 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7qnz\" (UniqueName: \"kubernetes.io/projected/87545d7c-46fa-44bf-af98-3e94095693ff-kube-api-access-b7qnz\") pod \"maas-api-revoke-keys-e2e-api-b-4266b7-p94mm\" (UID: \"87545d7c-46fa-44bf-af98-3e94095693ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4266b7-p94mm" Oct 30 21:24:39.449080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:39.449042 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87545d7c-46fa-44bf-af98-3e94095693ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4266b7-p94mm\" (UID: \"87545d7c-46fa-44bf-af98-3e94095693ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4266b7-p94mm" Oct 30 21:24:39.449157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:39.449143 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87545d7c-46fa-44bf-af98-3e94095693ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4266b7-p94mm\" (UID: \"87545d7c-46fa-44bf-af98-3e94095693ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4266b7-p94mm" Oct 30 21:24:39.550893 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:39.550855 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87545d7c-46fa-44bf-af98-3e94095693ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4266b7-p94mm\" (UID: \"87545d7c-46fa-44bf-af98-3e94095693ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4266b7-p94mm" Oct 30 21:24:39.551076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:39.550959 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87545d7c-46fa-44bf-af98-3e94095693ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4266b7-p94mm\" (UID: \"87545d7c-46fa-44bf-af98-3e94095693ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4266b7-p94mm" Oct 30 21:24:39.551126 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:39.551087 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7qnz\" (UniqueName: \"kubernetes.io/projected/87545d7c-46fa-44bf-af98-3e94095693ff-kube-api-access-b7qnz\") pod \"maas-api-revoke-keys-e2e-api-b-4266b7-p94mm\" (UID: \"87545d7c-46fa-44bf-af98-3e94095693ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4266b7-p94mm" Oct 30 21:24:39.551537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:39.551505 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87545d7c-46fa-44bf-af98-3e94095693ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4266b7-p94mm\" (UID: \"87545d7c-46fa-44bf-af98-3e94095693ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4266b7-p94mm" Oct 30 21:24:39.553342 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:39.553319 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87545d7c-46fa-44bf-af98-3e94095693ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4266b7-p94mm\" (UID: \"87545d7c-46fa-44bf-af98-3e94095693ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4266b7-p94mm" Oct 30 21:24:39.569910 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:39.569881 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7qnz\" (UniqueName: \"kubernetes.io/projected/87545d7c-46fa-44bf-af98-3e94095693ff-kube-api-access-b7qnz\") pod \"maas-api-revoke-keys-e2e-api-b-4266b7-p94mm\" (UID: \"87545d7c-46fa-44bf-af98-3e94095693ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4266b7-p94mm" Oct 30 21:24:39.676558 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:39.676523 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4266b7-p94mm" Oct 30 21:24:40.012927 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:24:40.012889 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87545d7c_46fa_44bf_af98_3e94095693ff.slice/crio-08adda58db963cd5fd99a218b780be0f31c55950cfc628fbc71285ec122cfa26 WatchSource:0}: Error finding container 08adda58db963cd5fd99a218b780be0f31c55950cfc628fbc71285ec122cfa26: Status 404 returned error can't find the container with id 08adda58db963cd5fd99a218b780be0f31c55950cfc628fbc71285ec122cfa26 Oct 30 21:24:40.016150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:40.016121 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4266b7-p94mm"] Oct 30 21:24:40.249216 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:40.248272 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4266b7-p94mm" event={"ID":"87545d7c-46fa-44bf-af98-3e94095693ff","Type":"ContainerStarted","Data":"08adda58db963cd5fd99a218b780be0f31c55950cfc628fbc71285ec122cfa26"} Oct 30 21:24:40.542649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:40.542565 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775"] Oct 30 21:24:40.542855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:40.542808 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" podUID="8c6b634e-2a0b-4c0a-aa07-7901e40d77db" containerName="istio-proxy" containerID="cri-o://e03d86a9106a4a248fde54f5b8774a3d0760a7099b452ef312a4961a588b7ac6" gracePeriod=30 Oct 30 21:24:41.138708 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:41.138674 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w"] Oct 30 21:24:41.143263 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:41.143244 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w" Oct 30 21:24:41.162719 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:41.162691 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w"] Oct 30 21:24:41.257512 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:41.257426 4926 generic.go:356] "Generic (PLEG): container finished" podID="87545d7c-46fa-44bf-af98-3e94095693ff" containerID="d0e69f971a6efb7df3561d55caed05dc887f59890bc8399b3991f3d6ece1352e" exitCode=0 Oct 30 21:24:41.257512 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:41.257478 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4266b7-p94mm" event={"ID":"87545d7c-46fa-44bf-af98-3e94095693ff","Type":"ContainerDied","Data":"d0e69f971a6efb7df3561d55caed05dc887f59890bc8399b3991f3d6ece1352e"} Oct 30 21:24:41.298227 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:41.298188 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df84efc-a402-4766-b7ad-60de74be2e0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w\" (UID: \"8df84efc-a402-4766-b7ad-60de74be2e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w" Oct 30 21:24:41.298405 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:41.298377 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm76h\" (UniqueName: \"kubernetes.io/projected/8df84efc-a402-4766-b7ad-60de74be2e0e-kube-api-access-zm76h\") pod \"maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w\" (UID: \"8df84efc-a402-4766-b7ad-60de74be2e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w" Oct 30 21:24:41.298451 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:41.298430 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df84efc-a402-4766-b7ad-60de74be2e0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w\" (UID: \"8df84efc-a402-4766-b7ad-60de74be2e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w" Oct 30 21:24:41.400297 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:41.400260 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df84efc-a402-4766-b7ad-60de74be2e0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w\" (UID: \"8df84efc-a402-4766-b7ad-60de74be2e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w" Oct 30 21:24:41.400484 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:41.400412 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm76h\" (UniqueName: \"kubernetes.io/projected/8df84efc-a402-4766-b7ad-60de74be2e0e-kube-api-access-zm76h\") pod \"maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w\" (UID: \"8df84efc-a402-4766-b7ad-60de74be2e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w" Oct 30 21:24:41.400484 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:41.400461 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df84efc-a402-4766-b7ad-60de74be2e0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w\" (UID: \"8df84efc-a402-4766-b7ad-60de74be2e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w" Oct 30 21:24:41.401164 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:41.401139 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df84efc-a402-4766-b7ad-60de74be2e0e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w\" (UID: \"8df84efc-a402-4766-b7ad-60de74be2e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w" Oct 30 21:24:41.403760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:41.403729 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df84efc-a402-4766-b7ad-60de74be2e0e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w\" (UID: \"8df84efc-a402-4766-b7ad-60de74be2e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w" Oct 30 21:24:41.420152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:41.420121 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm76h\" (UniqueName: \"kubernetes.io/projected/8df84efc-a402-4766-b7ad-60de74be2e0e-kube-api-access-zm76h\") pod \"maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w\" (UID: \"8df84efc-a402-4766-b7ad-60de74be2e0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w" Oct 30 21:24:41.457206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:41.457180 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w" Oct 30 21:24:41.972370 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:41.972336 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w"] Oct 30 21:24:41.977186 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:24:41.977157 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df84efc_a402_4766_b7ad_60de74be2e0e.slice/crio-a4c8e2ffe9e86f0564d5ea58ba44f7ce5cfff3f83703102eb7d3b7d033b4950b WatchSource:0}: Error finding container a4c8e2ffe9e86f0564d5ea58ba44f7ce5cfff3f83703102eb7d3b7d033b4950b: Status 404 returned error can't find the container with id a4c8e2ffe9e86f0564d5ea58ba44f7ce5cfff3f83703102eb7d3b7d033b4950b Oct 30 21:24:42.267852 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:42.267767 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w" event={"ID":"8df84efc-a402-4766-b7ad-60de74be2e0e","Type":"ContainerStarted","Data":"a4c8e2ffe9e86f0564d5ea58ba44f7ce5cfff3f83703102eb7d3b7d033b4950b"} Oct 30 21:24:42.444384 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:42.444355 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4266b7-p94mm" Oct 30 21:24:42.554114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:42.553672 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87545d7c-46fa-44bf-af98-3e94095693ff-kube-api-access-b7qnz\" (UniqueName: \"kubernetes.io/projected/87545d7c-46fa-44bf-af98-3e94095693ff-kube-api-access-b7qnz\") pod \"87545d7c-46fa-44bf-af98-3e94095693ff\" (UID: \"87545d7c-46fa-44bf-af98-3e94095693ff\") " Oct 30 21:24:42.554114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:42.553868 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87545d7c-46fa-44bf-af98-3e94095693ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87545d7c-46fa-44bf-af98-3e94095693ff-maas-api-openshift-ca\") pod \"87545d7c-46fa-44bf-af98-3e94095693ff\" (UID: \"87545d7c-46fa-44bf-af98-3e94095693ff\") " Oct 30 21:24:42.554362 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:42.554123 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87545d7c-46fa-44bf-af98-3e94095693ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87545d7c-46fa-44bf-af98-3e94095693ff-maas-api-xks-ca\") pod \"87545d7c-46fa-44bf-af98-3e94095693ff\" (UID: \"87545d7c-46fa-44bf-af98-3e94095693ff\") " Oct 30 21:24:42.572067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:42.559385 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87545d7c-46fa-44bf-af98-3e94095693ff-maas-api-openshift-ca" pod "87545d7c-46fa-44bf-af98-3e94095693ff" (UID: "87545d7c-46fa-44bf-af98-3e94095693ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:42.597054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:42.577000 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87545d7c-46fa-44bf-af98-3e94095693ff-maas-api-xks-ca" pod "87545d7c-46fa-44bf-af98-3e94095693ff" (UID: "87545d7c-46fa-44bf-af98-3e94095693ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:42.661294 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:42.659330 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87545d7c-46fa-44bf-af98-3e94095693ff-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:42.661294 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:42.659367 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87545d7c-46fa-44bf-af98-3e94095693ff-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:42.666698 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:42.666641 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87545d7c-46fa-44bf-af98-3e94095693ff-kube-api-access-b7qnz" pod "87545d7c-46fa-44bf-af98-3e94095693ff" (UID: "87545d7c-46fa-44bf-af98-3e94095693ff"). InnerVolumeSpecName "kube-api-access-b7qnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:42.761348 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:42.760991 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7qnz\" (UniqueName: \"kubernetes.io/projected/87545d7c-46fa-44bf-af98-3e94095693ff-kube-api-access-b7qnz\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:43.328253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:43.328165 4926 generic.go:356] "Generic (PLEG): container finished" podID="8df84efc-a402-4766-b7ad-60de74be2e0e" containerID="18d4353b7e18462cb185923e83a06f5095ef74ce0abd04576aabee15e0d552d1" exitCode=0 Oct 30 21:24:43.328742 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:43.328714 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w" event={"ID":"8df84efc-a402-4766-b7ad-60de74be2e0e","Type":"ContainerDied","Data":"18d4353b7e18462cb185923e83a06f5095ef74ce0abd04576aabee15e0d552d1"} Oct 30 21:24:43.330925 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:43.330835 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4266b7-p94mm" Oct 30 21:24:43.330925 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:43.330859 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4266b7-p94mm" event={"ID":"87545d7c-46fa-44bf-af98-3e94095693ff","Type":"ContainerDied","Data":"08adda58db963cd5fd99a218b780be0f31c55950cfc628fbc71285ec122cfa26"} Oct 30 21:24:43.330925 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:43.330903 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08adda58db963cd5fd99a218b780be0f31c55950cfc628fbc71285ec122cfa26" Oct 30 21:24:44.403069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:44.400135 4926 generic.go:356] "Generic (PLEG): container finished" podID="4aca75f2-2a47-484f-b19b-41c6d228cb2c" containerID="c2c8aa633a3df83d526b04a2ba33b53738de3747d6880c78ff7612d231275061" exitCode=0 Oct 30 21:24:44.403069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:44.400293 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" event={"ID":"4aca75f2-2a47-484f-b19b-41c6d228cb2c","Type":"ContainerDied","Data":"c2c8aa633a3df83d526b04a2ba33b53738de3747d6880c78ff7612d231275061"} Oct 30 21:24:44.403069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:44.400356 4926 scope.go:122] "RemoveContainer" containerID="dc884dc4bf22f01ff10c78a8c7cf11fde75968bea27bd1aec3e593014052384e" Oct 30 21:24:44.442982 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:44.441536 4926 generic.go:356] "Generic (PLEG): container finished" podID="3eddcba6-e6e3-49cd-9078-5777d0faff13" containerID="3ae73349604328c4e8f373d94b34f924fbedd2629a28e48e74e0d0c53683a4b4" exitCode=0 Oct 30 21:24:44.442982 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:44.442055 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" event={"ID":"3eddcba6-e6e3-49cd-9078-5777d0faff13","Type":"ContainerDied","Data":"3ae73349604328c4e8f373d94b34f924fbedd2629a28e48e74e0d0c53683a4b4"} Oct 30 21:24:44.853180 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:44.819130 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:24:44.892358 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:44.853130 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:24:44.939507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:44.939470 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:45.069597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.066573 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:45.069597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.069474 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-2tsvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:24:45.069597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.069525 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" podUID="217454ad-7acd-4b25-b91f-885bc464b6c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:45.161320 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.160896 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:24:45.441285 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.439180 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:24:45.444135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.442454 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-podinfo\") pod \"3eddcba6-e6e3-49cd-9078-5777d0faff13\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " Oct 30 21:24:45.444135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.442568 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-workload-certs\") pod \"3eddcba6-e6e3-49cd-9078-5777d0faff13\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " Oct 30 21:24:45.444135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.442648 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eddcba6-e6e3-49cd-9078-5777d0faff13-kube-api-access-wr74r\" (UniqueName: \"kubernetes.io/projected/3eddcba6-e6e3-49cd-9078-5777d0faff13-kube-api-access-wr74r\") pod \"3eddcba6-e6e3-49cd-9078-5777d0faff13\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " Oct 30 21:24:45.444135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.442711 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-envoy\") pod \"3eddcba6-e6e3-49cd-9078-5777d0faff13\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " Oct 30 21:24:45.444135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.442752 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-token\" (UniqueName: \"kubernetes.io/projected/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-token\") pod \"3eddcba6-e6e3-49cd-9078-5777d0faff13\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " Oct 30 21:24:45.444135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.442885 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eddcba6-e6e3-49cd-9078-5777d0faff13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eddcba6-e6e3-49cd-9078-5777d0faff13-istiod-ca-cert\") pod \"3eddcba6-e6e3-49cd-9078-5777d0faff13\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " Oct 30 21:24:45.444135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.442983 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-data\") pod \"3eddcba6-e6e3-49cd-9078-5777d0faff13\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " Oct 30 21:24:45.444135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.443057 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-credential-socket\") pod \"3eddcba6-e6e3-49cd-9078-5777d0faff13\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " Oct 30 21:24:45.444135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.443102 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-workload-socket\") pod \"3eddcba6-e6e3-49cd-9078-5777d0faff13\" (UID: \"3eddcba6-e6e3-49cd-9078-5777d0faff13\") " Oct 30 21:24:45.445221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.444813 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-workload-socket" pod "3eddcba6-e6e3-49cd-9078-5777d0faff13" (UID: "3eddcba6-e6e3-49cd-9078-5777d0faff13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:45.446054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.445903 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-workload-certs" pod "3eddcba6-e6e3-49cd-9078-5777d0faff13" (UID: "3eddcba6-e6e3-49cd-9078-5777d0faff13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:45.447187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.446914 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eddcba6-e6e3-49cd-9078-5777d0faff13-istiod-ca-cert" pod "3eddcba6-e6e3-49cd-9078-5777d0faff13" (UID: "3eddcba6-e6e3-49cd-9078-5777d0faff13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:45.477097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.463228 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-data" pod "3eddcba6-e6e3-49cd-9078-5777d0faff13" (UID: "3eddcba6-e6e3-49cd-9078-5777d0faff13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:45.477097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.472851 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-credential-socket" pod "3eddcba6-e6e3-49cd-9078-5777d0faff13" (UID: "3eddcba6-e6e3-49cd-9078-5777d0faff13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:45.511105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.500190 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-podinfo" pod "3eddcba6-e6e3-49cd-9078-5777d0faff13" (UID: "3eddcba6-e6e3-49cd-9078-5777d0faff13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:24:45.516897 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.516826 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-token" pod "3eddcba6-e6e3-49cd-9078-5777d0faff13" (UID: "3eddcba6-e6e3-49cd-9078-5777d0faff13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:45.547576 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.547538 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:45.547821 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.547801 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eddcba6-e6e3-49cd-9078-5777d0faff13-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:45.547965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.547953 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:45.548154 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.548095 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:45.548154 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.548113 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:45.548154 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.548127 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:45.548462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.548390 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:45.729178 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.729056 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eddcba6-e6e3-49cd-9078-5777d0faff13-kube-api-access-wr74r" pod "3eddcba6-e6e3-49cd-9078-5777d0faff13" (UID: "3eddcba6-e6e3-49cd-9078-5777d0faff13"). InnerVolumeSpecName "kube-api-access-wr74r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:45.759069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.758467 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" event={"ID":"3eddcba6-e6e3-49cd-9078-5777d0faff13","Type":"ContainerDied","Data":"622fc89a8e3bd1b1bd1fa7191b5d8b763ce5c5ef1cd5dc5dc6441a5aa59edf52"} Oct 30 21:24:45.759069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.758528 4926 scope.go:122] "RemoveContainer" containerID="3ae73349604328c4e8f373d94b34f924fbedd2629a28e48e74e0d0c53683a4b4" Oct 30 21:24:45.759069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.758661 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m" Oct 30 21:24:45.760100 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.759945 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr74r\" (UniqueName: \"kubernetes.io/projected/3eddcba6-e6e3-49cd-9078-5777d0faff13-kube-api-access-wr74r\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:45.762194 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.762173 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:24:45.763217 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.763178 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-envoy" pod "3eddcba6-e6e3-49cd-9078-5777d0faff13" (UID: "3eddcba6-e6e3-49cd-9078-5777d0faff13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:45.871121 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.871080 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aca75f2-2a47-484f-b19b-41c6d228cb2c-kube-api-access-9bnl5\" (UniqueName: \"kubernetes.io/projected/4aca75f2-2a47-484f-b19b-41c6d228cb2c-kube-api-access-9bnl5\") pod \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " Oct 30 21:24:45.880987 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.880948 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istiod-ca-cert\") pod \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " Oct 30 21:24:45.881166 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.881058 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-podinfo\") pod \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " Oct 30 21:24:45.881166 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.881110 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-data\") pod \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " Oct 30 21:24:45.881166 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.881158 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-credential-socket\") pod \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " Oct 30 21:24:45.881741 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.881712 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-token\" (UniqueName: \"kubernetes.io/projected/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-token\") pod \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " Oct 30 21:24:45.882022 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.881997 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-workload-socket\") pod \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " Oct 30 21:24:45.882248 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.882229 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-workload-certs\") pod \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " Oct 30 21:24:45.882582 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.882563 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-envoy\") pod \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\" (UID: \"4aca75f2-2a47-484f-b19b-41c6d228cb2c\") " Oct 30 21:24:45.882805 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.882773 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-workload-socket" pod "4aca75f2-2a47-484f-b19b-41c6d228cb2c" (UID: "4aca75f2-2a47-484f-b19b-41c6d228cb2c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:45.883066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.883019 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-workload-certs" pod "4aca75f2-2a47-484f-b19b-41c6d228cb2c" (UID: "4aca75f2-2a47-484f-b19b-41c6d228cb2c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:45.884256 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.884229 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-credential-socket" pod "4aca75f2-2a47-484f-b19b-41c6d228cb2c" (UID: "4aca75f2-2a47-484f-b19b-41c6d228cb2c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:45.885245 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.885136 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-data" pod "4aca75f2-2a47-484f-b19b-41c6d228cb2c" (UID: "4aca75f2-2a47-484f-b19b-41c6d228cb2c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:45.885245 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.885163 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:45.885245 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.885207 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:45.885245 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.885222 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eddcba6-e6e3-49cd-9078-5777d0faff13-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:45.885623 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.885452 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istiod-ca-cert" pod "4aca75f2-2a47-484f-b19b-41c6d228cb2c" (UID: "4aca75f2-2a47-484f-b19b-41c6d228cb2c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:45.887250 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.886800 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-podinfo" pod "4aca75f2-2a47-484f-b19b-41c6d228cb2c" (UID: "4aca75f2-2a47-484f-b19b-41c6d228cb2c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:24:45.888430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.888089 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-token" pod "4aca75f2-2a47-484f-b19b-41c6d228cb2c" (UID: "4aca75f2-2a47-484f-b19b-41c6d228cb2c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:45.888430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.888172 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aca75f2-2a47-484f-b19b-41c6d228cb2c-kube-api-access-9bnl5" pod "4aca75f2-2a47-484f-b19b-41c6d228cb2c" (UID: "4aca75f2-2a47-484f-b19b-41c6d228cb2c"). InnerVolumeSpecName "kube-api-access-9bnl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:45.888430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.888377 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-envoy" pod "4aca75f2-2a47-484f-b19b-41c6d228cb2c" (UID: "4aca75f2-2a47-484f-b19b-41c6d228cb2c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:45.909408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.909376 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w" Oct 30 21:24:45.986213 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.986136 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8df84efc-a402-4766-b7ad-60de74be2e0e-kube-api-access-zm76h\" (UniqueName: \"kubernetes.io/projected/8df84efc-a402-4766-b7ad-60de74be2e0e-kube-api-access-zm76h\") pod \"8df84efc-a402-4766-b7ad-60de74be2e0e\" (UID: \"8df84efc-a402-4766-b7ad-60de74be2e0e\") " Oct 30 21:24:45.986347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.986241 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8df84efc-a402-4766-b7ad-60de74be2e0e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df84efc-a402-4766-b7ad-60de74be2e0e-maas-api-openshift-ca\") pod \"8df84efc-a402-4766-b7ad-60de74be2e0e\" (UID: \"8df84efc-a402-4766-b7ad-60de74be2e0e\") " Oct 30 21:24:45.986442 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.986423 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8df84efc-a402-4766-b7ad-60de74be2e0e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df84efc-a402-4766-b7ad-60de74be2e0e-maas-api-xks-ca\") pod \"8df84efc-a402-4766-b7ad-60de74be2e0e\" (UID: \"8df84efc-a402-4766-b7ad-60de74be2e0e\") " Oct 30 21:24:45.986664 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.986635 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8df84efc-a402-4766-b7ad-60de74be2e0e-maas-api-openshift-ca" pod "8df84efc-a402-4766-b7ad-60de74be2e0e" (UID: "8df84efc-a402-4766-b7ad-60de74be2e0e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:45.987123 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.987102 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:45.987123 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.987123 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:45.987340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.987132 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8df84efc-a402-4766-b7ad-60de74be2e0e-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:45.987340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.987143 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:45.987340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.987152 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:45.987340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.987160 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:45.987340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.987167 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bnl5\" (UniqueName: \"kubernetes.io/projected/4aca75f2-2a47-484f-b19b-41c6d228cb2c-kube-api-access-9bnl5\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:45.987340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.987175 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aca75f2-2a47-484f-b19b-41c6d228cb2c-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:45.988307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.988277 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df84efc-a402-4766-b7ad-60de74be2e0e-kube-api-access-zm76h" pod "8df84efc-a402-4766-b7ad-60de74be2e0e" (UID: "8df84efc-a402-4766-b7ad-60de74be2e0e"). InnerVolumeSpecName "kube-api-access-zm76h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:45.988620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:45.988598 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8df84efc-a402-4766-b7ad-60de74be2e0e-maas-api-xks-ca" pod "8df84efc-a402-4766-b7ad-60de74be2e0e" (UID: "8df84efc-a402-4766-b7ad-60de74be2e0e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:24:46.035675 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.035345 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:24:46.091294 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.091258 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8df84efc-a402-4766-b7ad-60de74be2e0e-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:46.097776 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.097736 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm76h\" (UniqueName: \"kubernetes.io/projected/8df84efc-a402-4766-b7ad-60de74be2e0e-kube-api-access-zm76h\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:46.133367 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.129557 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m"] Oct 30 21:24:46.142077 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.142027 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-xwn5m"] Oct 30 21:24:46.199228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.199185 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istiod-ca-cert\") pod \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " Oct 30 21:24:46.199433 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.199273 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-workload-socket\") pod \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " Oct 30 21:24:46.199511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.199445 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-data\") pod \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " Oct 30 21:24:46.199573 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.199510 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-envoy\") pod \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " Oct 30 21:24:46.199573 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.199536 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-credential-socket\") pod \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " Oct 30 21:24:46.199733 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.199628 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-kube-api-access-lrqcs\" (UniqueName: \"kubernetes.io/projected/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-kube-api-access-lrqcs\") pod \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " Oct 30 21:24:46.199733 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.199653 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-workload-certs\") pod \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " Oct 30 21:24:46.199733 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.199715 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-podinfo\") pod \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " Oct 30 21:24:46.199932 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.199802 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-token\" (UniqueName: \"kubernetes.io/projected/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-token\") pod \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\" (UID: \"8c6b634e-2a0b-4c0a-aa07-7901e40d77db\") " Oct 30 21:24:46.201847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.201565 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-data" pod "8c6b634e-2a0b-4c0a-aa07-7901e40d77db" (UID: "8c6b634e-2a0b-4c0a-aa07-7901e40d77db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:46.201972 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.201949 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istiod-ca-cert" pod "8c6b634e-2a0b-4c0a-aa07-7901e40d77db" (UID: "8c6b634e-2a0b-4c0a-aa07-7901e40d77db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:24:46.202273 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.202253 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-workload-socket" pod "8c6b634e-2a0b-4c0a-aa07-7901e40d77db" (UID: "8c6b634e-2a0b-4c0a-aa07-7901e40d77db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:46.202693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.202603 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-credential-socket" pod "8c6b634e-2a0b-4c0a-aa07-7901e40d77db" (UID: "8c6b634e-2a0b-4c0a-aa07-7901e40d77db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:46.202838 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.202792 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-workload-certs" pod "8c6b634e-2a0b-4c0a-aa07-7901e40d77db" (UID: "8c6b634e-2a0b-4c0a-aa07-7901e40d77db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:46.214390 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.214348 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-envoy" pod "8c6b634e-2a0b-4c0a-aa07-7901e40d77db" (UID: "8c6b634e-2a0b-4c0a-aa07-7901e40d77db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:24:46.215363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.215329 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-token" pod "8c6b634e-2a0b-4c0a-aa07-7901e40d77db" (UID: "8c6b634e-2a0b-4c0a-aa07-7901e40d77db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:46.222724 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.222693 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-kube-api-access-lrqcs" pod "8c6b634e-2a0b-4c0a-aa07-7901e40d77db" (UID: "8c6b634e-2a0b-4c0a-aa07-7901e40d77db"). InnerVolumeSpecName "kube-api-access-lrqcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:24:46.223714 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.223659 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-podinfo" pod "8c6b634e-2a0b-4c0a-aa07-7901e40d77db" (UID: "8c6b634e-2a0b-4c0a-aa07-7901e40d77db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:24:46.302257 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.302174 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:46.302257 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.302209 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:46.302257 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.302224 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:46.302257 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.302238 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:46.302257 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.302252 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:46.302537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.302265 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:46.302537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.302279 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrqcs\" (UniqueName: \"kubernetes.io/projected/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-kube-api-access-lrqcs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:46.302537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.302296 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:46.302537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.302309 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c6b634e-2a0b-4c0a-aa07-7901e40d77db-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:24:46.773863 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.773818 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" event={"ID":"4aca75f2-2a47-484f-b19b-41c6d228cb2c","Type":"ContainerDied","Data":"dc12adc1a0c57ce5ee41ce0f2796973872c00d233ad27124e04d9df92ded7cc6"} Oct 30 21:24:46.774461 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.773928 4926 scope.go:122] "RemoveContainer" containerID="c2c8aa633a3df83d526b04a2ba33b53738de3747d6880c78ff7612d231275061" Oct 30 21:24:46.774461 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.774151 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx" Oct 30 21:24:46.780374 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.780338 4926 generic.go:356] "Generic (PLEG): container finished" podID="8c6b634e-2a0b-4c0a-aa07-7901e40d77db" containerID="e03d86a9106a4a248fde54f5b8774a3d0760a7099b452ef312a4961a588b7ac6" exitCode=0 Oct 30 21:24:46.780528 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.780429 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" event={"ID":"8c6b634e-2a0b-4c0a-aa07-7901e40d77db","Type":"ContainerDied","Data":"e03d86a9106a4a248fde54f5b8774a3d0760a7099b452ef312a4961a588b7ac6"} Oct 30 21:24:46.780528 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.780482 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" event={"ID":"8c6b634e-2a0b-4c0a-aa07-7901e40d77db","Type":"ContainerDied","Data":"d32e9aa19b0f8423e118025adaac232238433501faaa5a771bdc71758268a4da"} Oct 30 21:24:46.780655 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.780592 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775" Oct 30 21:24:46.785045 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.784994 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w" event={"ID":"8df84efc-a402-4766-b7ad-60de74be2e0e","Type":"ContainerDied","Data":"a4c8e2ffe9e86f0564d5ea58ba44f7ce5cfff3f83703102eb7d3b7d033b4950b"} Oct 30 21:24:46.785156 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.785071 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4c8e2ffe9e86f0564d5ea58ba44f7ce5cfff3f83703102eb7d3b7d033b4950b" Oct 30 21:24:46.785156 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.785146 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w" Oct 30 21:24:46.806651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.806628 4926 scope.go:122] "RemoveContainer" containerID="e03d86a9106a4a248fde54f5b8774a3d0760a7099b452ef312a4961a588b7ac6" Oct 30 21:24:46.818760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.818739 4926 scope.go:122] "RemoveContainer" containerID="140ab3d0fb774e808bd48d46b3bc9e9165024f96dc38bd16ca136012d9f4c481" Oct 30 21:24:46.830797 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.830773 4926 scope.go:122] "RemoveContainer" containerID="e03d86a9106a4a248fde54f5b8774a3d0760a7099b452ef312a4961a588b7ac6" Oct 30 21:24:46.831180 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:24:46.831158 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e03d86a9106a4a248fde54f5b8774a3d0760a7099b452ef312a4961a588b7ac6\": container with ID starting with e03d86a9106a4a248fde54f5b8774a3d0760a7099b452ef312a4961a588b7ac6 not found: ID does not exist" containerID="e03d86a9106a4a248fde54f5b8774a3d0760a7099b452ef312a4961a588b7ac6" Oct 30 21:24:46.831254 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.831192 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e03d86a9106a4a248fde54f5b8774a3d0760a7099b452ef312a4961a588b7ac6"} err="failed to get container status \"e03d86a9106a4a248fde54f5b8774a3d0760a7099b452ef312a4961a588b7ac6\": rpc error: code = NotFound desc = could not find container \"e03d86a9106a4a248fde54f5b8774a3d0760a7099b452ef312a4961a588b7ac6\": container with ID starting with e03d86a9106a4a248fde54f5b8774a3d0760a7099b452ef312a4961a588b7ac6 not found: ID does not exist" Oct 30 21:24:46.831254 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.831210 4926 scope.go:122] "RemoveContainer" containerID="140ab3d0fb774e808bd48d46b3bc9e9165024f96dc38bd16ca136012d9f4c481" Oct 30 21:24:46.831538 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:24:46.831517 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"140ab3d0fb774e808bd48d46b3bc9e9165024f96dc38bd16ca136012d9f4c481\": container with ID starting with 140ab3d0fb774e808bd48d46b3bc9e9165024f96dc38bd16ca136012d9f4c481 not found: ID does not exist" containerID="140ab3d0fb774e808bd48d46b3bc9e9165024f96dc38bd16ca136012d9f4c481" Oct 30 21:24:46.831603 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.831546 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"140ab3d0fb774e808bd48d46b3bc9e9165024f96dc38bd16ca136012d9f4c481"} err="failed to get container status \"140ab3d0fb774e808bd48d46b3bc9e9165024f96dc38bd16ca136012d9f4c481\": rpc error: code = NotFound desc = could not find container \"140ab3d0fb774e808bd48d46b3bc9e9165024f96dc38bd16ca136012d9f4c481\": container with ID starting with 140ab3d0fb774e808bd48d46b3bc9e9165024f96dc38bd16ca136012d9f4c481 not found: ID does not exist" Oct 30 21:24:46.844979 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.844950 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775"] Oct 30 21:24:46.857076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.857044 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0e7510-openshift-default-5d8c44ffd9-zl775"] Oct 30 21:24:46.880749 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.880714 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx"] Oct 30 21:24:46.896092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:46.894403 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-59fb17-openshift-default-84d55cd8fc-c96xx"] Oct 30 21:24:47.021570 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:47.021535 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eddcba6-e6e3-49cd-9078-5777d0faff13" path="/var/lib/kubelet/pods/3eddcba6-e6e3-49cd-9078-5777d0faff13/volumes" Oct 30 21:24:47.022379 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:47.022351 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aca75f2-2a47-484f-b19b-41c6d228cb2c" path="/var/lib/kubelet/pods/4aca75f2-2a47-484f-b19b-41c6d228cb2c/volumes" Oct 30 21:24:47.023143 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:24:47.023123 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c6b634e-2a0b-4c0a-aa07-7901e40d77db" path="/var/lib/kubelet/pods/8c6b634e-2a0b-4c0a-aa07-7901e40d77db/volumes" Oct 30 21:25:06.454023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.453989 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7"] Oct 30 21:25:06.454615 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.454600 4926 cpu_manager.go:414] "containerMap: removing container" podUID="3eddcba6-e6e3-49cd-9078-5777d0faff13" containerName="istio-proxy" Oct 30 21:25:06.454663 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.454616 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eddcba6-e6e3-49cd-9078-5777d0faff13" containerName="istio-proxy" Oct 30 21:25:06.454663 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.454626 4926 cpu_manager.go:414] "containerMap: removing container" podUID="8c6b634e-2a0b-4c0a-aa07-7901e40d77db" containerName="istio-proxy" Oct 30 21:25:06.454663 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.454632 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c6b634e-2a0b-4c0a-aa07-7901e40d77db" containerName="istio-proxy" Oct 30 21:25:06.454663 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.454651 4926 cpu_manager.go:414] "containerMap: removing container" podUID="4aca75f2-2a47-484f-b19b-41c6d228cb2c" containerName="istio-proxy" Oct 30 21:25:06.454663 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.454656 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aca75f2-2a47-484f-b19b-41c6d228cb2c" containerName="istio-proxy" Oct 30 21:25:06.454822 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.454670 4926 cpu_manager.go:414] "containerMap: removing container" podUID="4aca75f2-2a47-484f-b19b-41c6d228cb2c" containerName="istio-proxy" Oct 30 21:25:06.454822 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.454675 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aca75f2-2a47-484f-b19b-41c6d228cb2c" containerName="istio-proxy" Oct 30 21:25:06.454822 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.454684 4926 cpu_manager.go:414] "containerMap: removing container" podUID="87545d7c-46fa-44bf-af98-3e94095693ff" containerName="revoke-keys" Oct 30 21:25:06.454822 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.454689 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87545d7c-46fa-44bf-af98-3e94095693ff" containerName="revoke-keys" Oct 30 21:25:06.454822 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.454694 4926 cpu_manager.go:414] "containerMap: removing container" podUID="8df84efc-a402-4766-b7ad-60de74be2e0e" containerName="revoke-keys" Oct 30 21:25:06.454822 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.454699 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df84efc-a402-4766-b7ad-60de74be2e0e" containerName="revoke-keys" Oct 30 21:25:06.454822 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.454815 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eddcba6-e6e3-49cd-9078-5777d0faff13" containerName="istio-proxy" Oct 30 21:25:06.455057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.454828 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c6b634e-2a0b-4c0a-aa07-7901e40d77db" containerName="istio-proxy" Oct 30 21:25:06.455057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.454837 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aca75f2-2a47-484f-b19b-41c6d228cb2c" containerName="istio-proxy" Oct 30 21:25:06.455057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.454842 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c6b634e-2a0b-4c0a-aa07-7901e40d77db" containerName="istio-proxy" Oct 30 21:25:06.455057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.454851 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="87545d7c-46fa-44bf-af98-3e94095693ff" containerName="revoke-keys" Oct 30 21:25:06.455057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.454857 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df84efc-a402-4766-b7ad-60de74be2e0e" containerName="revoke-keys" Oct 30 21:25:06.461328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.460204 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7" Oct 30 21:25:06.477562 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.477527 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7"] Oct 30 21:25:06.481612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.481584 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:25:06.481755 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.481677 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:25:06.570470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.570422 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/158962c8-8c25-4dc6-8c5f-31b330c15f96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1da2183-t87p7\" (UID: \"158962c8-8c25-4dc6-8c5f-31b330c15f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7" Oct 30 21:25:06.570677 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.570582 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/158962c8-8c25-4dc6-8c5f-31b330c15f96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1da2183-t87p7\" (UID: \"158962c8-8c25-4dc6-8c5f-31b330c15f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7" Oct 30 21:25:06.570830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.570802 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gsl7\" (UniqueName: \"kubernetes.io/projected/158962c8-8c25-4dc6-8c5f-31b330c15f96-kube-api-access-6gsl7\") pod \"maas-api-revoke-keys-e2e-mt-c1da2183-t87p7\" (UID: \"158962c8-8c25-4dc6-8c5f-31b330c15f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7" Oct 30 21:25:06.672148 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.672111 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gsl7\" (UniqueName: \"kubernetes.io/projected/158962c8-8c25-4dc6-8c5f-31b330c15f96-kube-api-access-6gsl7\") pod \"maas-api-revoke-keys-e2e-mt-c1da2183-t87p7\" (UID: \"158962c8-8c25-4dc6-8c5f-31b330c15f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7" Oct 30 21:25:06.672358 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.672269 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/158962c8-8c25-4dc6-8c5f-31b330c15f96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1da2183-t87p7\" (UID: \"158962c8-8c25-4dc6-8c5f-31b330c15f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7" Oct 30 21:25:06.672436 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.672369 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/158962c8-8c25-4dc6-8c5f-31b330c15f96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1da2183-t87p7\" (UID: \"158962c8-8c25-4dc6-8c5f-31b330c15f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7" Oct 30 21:25:06.673101 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.673066 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/158962c8-8c25-4dc6-8c5f-31b330c15f96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1da2183-t87p7\" (UID: \"158962c8-8c25-4dc6-8c5f-31b330c15f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7" Oct 30 21:25:06.674704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.674678 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/158962c8-8c25-4dc6-8c5f-31b330c15f96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-c1da2183-t87p7\" (UID: \"158962c8-8c25-4dc6-8c5f-31b330c15f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7" Oct 30 21:25:06.690727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.690700 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gsl7\" (UniqueName: \"kubernetes.io/projected/158962c8-8c25-4dc6-8c5f-31b330c15f96-kube-api-access-6gsl7\") pod \"maas-api-revoke-keys-e2e-mt-c1da2183-t87p7\" (UID: \"158962c8-8c25-4dc6-8c5f-31b330c15f96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7" Oct 30 21:25:06.781719 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:06.781621 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7" Oct 30 21:25:07.100887 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:25:07.100856 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158962c8_8c25_4dc6_8c5f_31b330c15f96.slice/crio-fadd5b506f33658ce16396f633c20e957c2b0c4afe5d940ab368f47ab5e54cb3 WatchSource:0}: Error finding container fadd5b506f33658ce16396f633c20e957c2b0c4afe5d940ab368f47ab5e54cb3: Status 404 returned error can't find the container with id fadd5b506f33658ce16396f633c20e957c2b0c4afe5d940ab368f47ab5e54cb3 Oct 30 21:25:07.102002 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.101891 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7"] Oct 30 21:25:07.125758 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.125725 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj"] Oct 30 21:25:07.126760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.126734 4926 cpu_manager.go:414] "containerMap: removing container" podUID="8c6b634e-2a0b-4c0a-aa07-7901e40d77db" containerName="istio-proxy" Oct 30 21:25:07.126760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.126760 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c6b634e-2a0b-4c0a-aa07-7901e40d77db" containerName="istio-proxy" Oct 30 21:25:07.126951 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.126937 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aca75f2-2a47-484f-b19b-41c6d228cb2c" containerName="istio-proxy" Oct 30 21:25:07.140743 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.140716 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.143221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.143193 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj"] Oct 30 21:25:07.159178 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.158845 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1eab65-openshift-default-dockercfg-5jsjw\"" Oct 30 21:25:07.181472 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.181437 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-token\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.181826 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.181776 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-data\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.182008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.181988 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgps2\" (UniqueName: \"kubernetes.io/projected/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-kube-api-access-pgps2\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.182227 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.182209 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-credential-socket\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.182378 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.182360 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-workload-certs\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.182533 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.182516 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istiod-ca-cert\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.182716 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.182698 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-envoy\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.182931 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.182913 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-podinfo\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.183115 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.183096 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-workload-socket\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.322056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.317951 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-token\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.322056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.318060 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-data\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.322056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.318127 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgps2\" (UniqueName: \"kubernetes.io/projected/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-kube-api-access-pgps2\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.322056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.318228 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-credential-socket\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.322056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.318295 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-workload-certs\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.322056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.318376 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istiod-ca-cert\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.322056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.318500 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-envoy\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.322056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.318655 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-podinfo\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.322056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.318751 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-workload-socket\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.322056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.319009 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-data\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.322056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.319281 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-workload-socket\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.322056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.319735 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istiod-ca-cert\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.332064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.325472 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-credential-socket\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.332064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.326598 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-workload-certs\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.342859 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.342829 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgps2\" (UniqueName: \"kubernetes.io/projected/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-kube-api-access-pgps2\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.348102 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.347518 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-token\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.348102 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.347824 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-envoy\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.359344 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.354632 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-podinfo\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.467255 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.466971 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:07.514139 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:25:07.514081 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:25:07.838315 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.837358 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj"] Oct 30 21:25:07.875525 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.871309 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:25:07.875525 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.871407 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:25:07.875525 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:07.871451 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:25:08.011292 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:08.011252 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7" event={"ID":"158962c8-8c25-4dc6-8c5f-31b330c15f96","Type":"ContainerStarted","Data":"2d8ac12ee91f4ba33bb77b725247c0959814c9bb53cf580bab59c177a45ace35"} Oct 30 21:25:08.011505 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:08.011303 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7" event={"ID":"158962c8-8c25-4dc6-8c5f-31b330c15f96","Type":"ContainerStarted","Data":"fadd5b506f33658ce16396f633c20e957c2b0c4afe5d940ab368f47ab5e54cb3"} Oct 30 21:25:08.017122 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:08.016379 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" event={"ID":"76bddd4e-d53f-41f6-95a5-bc445d3eedcf","Type":"ContainerStarted","Data":"a4e7592dd1ff4fbb169797f585775c45b5a7b6e2b55bd3a64890ee48f2d691a0"} Oct 30 21:25:08.017122 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:08.016435 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" event={"ID":"76bddd4e-d53f-41f6-95a5-bc445d3eedcf","Type":"ContainerStarted","Data":"5e5c70cf204f4903ab5668b4d00f119aa3978ea2a761840f0b4af3c30a8f7f28"} Oct 30 21:25:08.052614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:08.052561 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7" podStartSLOduration=2.052547955 podStartE2EDuration="2.052547955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:08.049255384 +0000 UTC m=+1495.170329131" watchObservedRunningTime="2026-10-30 21:25:08.052547955 +0000 UTC m=+1495.173621701" Oct 30 21:25:08.468053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:08.467996 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:08.471108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:08.471070 4926 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:25:08.471264 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:08.471132 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" podUID="76bddd4e-d53f-41f6-95a5-bc445d3eedcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 21:25:09.028147 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:09.028111 4926 generic.go:356] "Generic (PLEG): container finished" podID="158962c8-8c25-4dc6-8c5f-31b330c15f96" containerID="2d8ac12ee91f4ba33bb77b725247c0959814c9bb53cf580bab59c177a45ace35" exitCode=0 Oct 30 21:25:09.028355 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:09.028197 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7" event={"ID":"158962c8-8c25-4dc6-8c5f-31b330c15f96","Type":"ContainerDied","Data":"2d8ac12ee91f4ba33bb77b725247c0959814c9bb53cf580bab59c177a45ace35"} Oct 30 21:25:09.069429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:09.069373 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" podStartSLOduration=2.069358814 podStartE2EDuration="2.069358814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:08.094267415 +0000 UTC m=+1495.215341163" watchObservedRunningTime="2026-10-30 21:25:09.069358814 +0000 UTC m=+1496.190432558" Oct 30 21:25:09.472844 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:09.472811 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:10.036386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:10.036351 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:10.037375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:10.037350 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:25:10.274845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:10.274808 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7" Oct 30 21:25:10.390233 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:10.390139 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158962c8-8c25-4dc6-8c5f-31b330c15f96-kube-api-access-6gsl7\" (UniqueName: \"kubernetes.io/projected/158962c8-8c25-4dc6-8c5f-31b330c15f96-kube-api-access-6gsl7\") pod \"158962c8-8c25-4dc6-8c5f-31b330c15f96\" (UID: \"158962c8-8c25-4dc6-8c5f-31b330c15f96\") " Oct 30 21:25:10.390406 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:10.390258 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/158962c8-8c25-4dc6-8c5f-31b330c15f96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/158962c8-8c25-4dc6-8c5f-31b330c15f96-maas-api-openshift-ca\") pod \"158962c8-8c25-4dc6-8c5f-31b330c15f96\" (UID: \"158962c8-8c25-4dc6-8c5f-31b330c15f96\") " Oct 30 21:25:10.390671 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:10.390645 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158962c8-8c25-4dc6-8c5f-31b330c15f96-maas-api-openshift-ca" pod "158962c8-8c25-4dc6-8c5f-31b330c15f96" (UID: "158962c8-8c25-4dc6-8c5f-31b330c15f96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:10.391089 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:10.391065 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/158962c8-8c25-4dc6-8c5f-31b330c15f96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/158962c8-8c25-4dc6-8c5f-31b330c15f96-maas-api-xks-ca\") pod \"158962c8-8c25-4dc6-8c5f-31b330c15f96\" (UID: \"158962c8-8c25-4dc6-8c5f-31b330c15f96\") " Oct 30 21:25:10.392210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:10.392185 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/158962c8-8c25-4dc6-8c5f-31b330c15f96-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:25:10.392866 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:10.392822 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158962c8-8c25-4dc6-8c5f-31b330c15f96-kube-api-access-6gsl7" pod "158962c8-8c25-4dc6-8c5f-31b330c15f96" (UID: "158962c8-8c25-4dc6-8c5f-31b330c15f96"). InnerVolumeSpecName "kube-api-access-6gsl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:10.398189 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:10.398163 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/158962c8-8c25-4dc6-8c5f-31b330c15f96-maas-api-xks-ca" pod "158962c8-8c25-4dc6-8c5f-31b330c15f96" (UID: "158962c8-8c25-4dc6-8c5f-31b330c15f96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:25:10.494043 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:10.494000 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/158962c8-8c25-4dc6-8c5f-31b330c15f96-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:25:10.494445 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:10.494058 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gsl7\" (UniqueName: \"kubernetes.io/projected/158962c8-8c25-4dc6-8c5f-31b330c15f96-kube-api-access-6gsl7\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:25:10.572758 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:10.572722 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:25:11.046182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:11.046154 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7" Oct 30 21:25:11.046182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:11.046168 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7" event={"ID":"158962c8-8c25-4dc6-8c5f-31b330c15f96","Type":"ContainerDied","Data":"fadd5b506f33658ce16396f633c20e957c2b0c4afe5d940ab368f47ab5e54cb3"} Oct 30 21:25:11.046407 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:11.046226 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fadd5b506f33658ce16396f633c20e957c2b0c4afe5d940ab368f47ab5e54cb3" Oct 30 21:25:13.353097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:13.353067 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:25:13.358075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:13.358005 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:25:15.029374 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:15.029345 4926 scope.go:122] "RemoveContainer" containerID="af4d55d33e63f8e49c87237ab6576a9282d90a72e8a090171724a2e4dd3fd04b" Oct 30 21:25:21.968659 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:21.968624 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8"] Oct 30 21:25:21.969424 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:21.969403 4926 cpu_manager.go:414] "containerMap: removing container" podUID="158962c8-8c25-4dc6-8c5f-31b330c15f96" containerName="revoke-keys" Oct 30 21:25:21.969493 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:21.969425 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158962c8-8c25-4dc6-8c5f-31b330c15f96" containerName="revoke-keys" Oct 30 21:25:21.969566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:21.969556 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="158962c8-8c25-4dc6-8c5f-31b330c15f96" containerName="revoke-keys" Oct 30 21:25:21.974995 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:21.974972 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.006399 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.006372 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8"] Oct 30 21:25:22.010338 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.010310 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c31999-openshift-default-dockercfg-j7gl5\"" Oct 30 21:25:22.015977 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.015943 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmdsj\" (UniqueName: \"kubernetes.io/projected/ebd67b20-66f0-472a-ac9b-43aa93a91264-kube-api-access-lmdsj\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.016151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.015997 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-workload-certs\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.016151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.016024 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-data\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.016151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.016089 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-token\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.016151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.016136 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd67b20-66f0-472a-ac9b-43aa93a91264-istiod-ca-cert\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.016151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.016151 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-podinfo\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.016419 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.016211 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-envoy\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.016419 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.016230 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-workload-socket\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.016419 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.016266 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-credential-socket\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.117277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.117241 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-envoy\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.117472 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.117295 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-workload-socket\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.117472 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.117337 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-credential-socket\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.117623 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.117603 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmdsj\" (UniqueName: \"kubernetes.io/projected/ebd67b20-66f0-472a-ac9b-43aa93a91264-kube-api-access-lmdsj\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.117668 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.117655 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-workload-certs\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.117704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.117688 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-data\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.117772 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.117743 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-credential-socket\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.117842 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.117790 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-token\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.117992 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.117964 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd67b20-66f0-472a-ac9b-43aa93a91264-istiod-ca-cert\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.118121 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.118011 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-data\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.118224 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.118201 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-podinfo\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.118291 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.118272 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-workload-certs\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.118706 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.118676 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd67b20-66f0-472a-ac9b-43aa93a91264-istiod-ca-cert\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.118830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.118724 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-workload-socket\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.120134 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.120111 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-envoy\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.120709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.120687 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-podinfo\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.135948 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.135924 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-token\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.136063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.135924 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmdsj\" (UniqueName: \"kubernetes.io/projected/ebd67b20-66f0-472a-ac9b-43aa93a91264-kube-api-access-lmdsj\") pod \"e2e-inf-a-c31999-openshift-default-967c87455-pshs8\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.291101 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.290126 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:22.684301 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.684254 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8"] Oct 30 21:25:22.684590 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:25:22.684566 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd67b20_66f0_472a_ac9b_43aa93a91264.slice/crio-c94e2fff8a71f4fbd4a6b2b38ca25b2382392206d61dad0658f19bbe7b0de10d WatchSource:0}: Error finding container c94e2fff8a71f4fbd4a6b2b38ca25b2382392206d61dad0658f19bbe7b0de10d: Status 404 returned error can't find the container with id c94e2fff8a71f4fbd4a6b2b38ca25b2382392206d61dad0658f19bbe7b0de10d Oct 30 21:25:22.688323 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.688289 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:25:22.688424 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.688362 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:25:22.688424 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:22.688392 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:25:23.191917 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:23.191876 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" event={"ID":"ebd67b20-66f0-472a-ac9b-43aa93a91264","Type":"ContainerStarted","Data":"7b41130fe3fd8a3263c6b20eb51bd813e90434097fe9bf7020e2b70fe942dbb6"} Oct 30 21:25:23.192535 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:23.191949 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" event={"ID":"ebd67b20-66f0-472a-ac9b-43aa93a91264","Type":"ContainerStarted","Data":"c94e2fff8a71f4fbd4a6b2b38ca25b2382392206d61dad0658f19bbe7b0de10d"} Oct 30 21:25:23.245325 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:23.245259 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" podStartSLOduration=2.245239973 podStartE2EDuration="2.245239973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:23.243085347 +0000 UTC m=+1510.364159095" watchObservedRunningTime="2026-10-30 21:25:23.245239973 +0000 UTC m=+1510.366313720" Oct 30 21:25:23.298173 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:23.298090 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:23.306882 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:23.306633 4926 patch_prober.go:29] interesting pod/e2e-inf-a-c31999-openshift-default-967c87455-pshs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:25:23.306882 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:23.306691 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" podUID="ebd67b20-66f0-472a-ac9b-43aa93a91264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 21:25:24.298058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:24.293465 4926 patch_prober.go:29] interesting pod/e2e-inf-a-c31999-openshift-default-967c87455-pshs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:25:24.298058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:24.293531 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" podUID="ebd67b20-66f0-472a-ac9b-43aa93a91264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 21:25:25.294455 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:25.294415 4926 patch_prober.go:29] interesting pod/e2e-inf-a-c31999-openshift-default-967c87455-pshs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:25:25.294737 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:25.294709 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" podUID="ebd67b20-66f0-472a-ac9b-43aa93a91264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 21:25:26.310056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:26.290473 4926 patch_prober.go:29] interesting pod/e2e-inf-a-c31999-openshift-default-967c87455-pshs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:25:26.310056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:26.290544 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" podUID="ebd67b20-66f0-472a-ac9b-43aa93a91264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 21:25:27.298417 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:27.298382 4926 patch_prober.go:29] interesting pod/e2e-inf-a-c31999-openshift-default-967c87455-pshs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:25:27.298954 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:27.298843 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" podUID="ebd67b20-66f0-472a-ac9b-43aa93a91264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 21:25:28.294055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:28.293124 4926 patch_prober.go:29] interesting pod/e2e-inf-a-c31999-openshift-default-967c87455-pshs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:25:28.294055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:28.293186 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" podUID="ebd67b20-66f0-472a-ac9b-43aa93a91264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 21:25:29.290819 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:29.290776 4926 patch_prober.go:29] interesting pod/e2e-inf-a-c31999-openshift-default-967c87455-pshs8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body= Oct 30 21:25:29.291147 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:29.291111 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" podUID="ebd67b20-66f0-472a-ac9b-43aa93a91264" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" Oct 30 21:25:30.296219 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:30.296184 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:30.296711 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:30.296691 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:30.297525 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:30.297503 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:25:31.227492 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:31.227459 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j"] Oct 30 21:25:31.232173 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:31.232149 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j" Oct 30 21:25:31.247392 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:31.247373 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:25:31.257887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:31.257868 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:25:31.281839 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:31.281814 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j"] Oct 30 21:25:31.313467 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:31.313436 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99af3b4-3d20-4bb2-a59a-712fc18e35f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j\" (UID: \"a99af3b4-3d20-4bb2-a59a-712fc18e35f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j" Oct 30 21:25:31.313859 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:31.313647 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5hsp\" (UniqueName: \"kubernetes.io/projected/a99af3b4-3d20-4bb2-a59a-712fc18e35f8-kube-api-access-c5hsp\") pod \"maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j\" (UID: \"a99af3b4-3d20-4bb2-a59a-712fc18e35f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j" Oct 30 21:25:31.313859 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:31.313693 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99af3b4-3d20-4bb2-a59a-712fc18e35f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j\" (UID: \"a99af3b4-3d20-4bb2-a59a-712fc18e35f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j" Oct 30 21:25:31.415154 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:31.415119 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99af3b4-3d20-4bb2-a59a-712fc18e35f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j\" (UID: \"a99af3b4-3d20-4bb2-a59a-712fc18e35f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j" Oct 30 21:25:31.415330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:31.415282 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5hsp\" (UniqueName: \"kubernetes.io/projected/a99af3b4-3d20-4bb2-a59a-712fc18e35f8-kube-api-access-c5hsp\") pod \"maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j\" (UID: \"a99af3b4-3d20-4bb2-a59a-712fc18e35f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j" Oct 30 21:25:31.415330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:31.415318 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99af3b4-3d20-4bb2-a59a-712fc18e35f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j\" (UID: \"a99af3b4-3d20-4bb2-a59a-712fc18e35f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j" Oct 30 21:25:31.415996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:31.415970 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99af3b4-3d20-4bb2-a59a-712fc18e35f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j\" (UID: \"a99af3b4-3d20-4bb2-a59a-712fc18e35f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j" Oct 30 21:25:31.418020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:31.417995 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99af3b4-3d20-4bb2-a59a-712fc18e35f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j\" (UID: \"a99af3b4-3d20-4bb2-a59a-712fc18e35f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j" Oct 30 21:25:31.442213 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:31.442191 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5hsp\" (UniqueName: \"kubernetes.io/projected/a99af3b4-3d20-4bb2-a59a-712fc18e35f8-kube-api-access-c5hsp\") pod \"maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j\" (UID: \"a99af3b4-3d20-4bb2-a59a-712fc18e35f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j" Oct 30 21:25:31.547535 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:31.547460 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j" Oct 30 21:25:31.856503 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:31.856464 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j"] Oct 30 21:25:31.857985 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:25:31.857961 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99af3b4_3d20_4bb2_a59a_712fc18e35f8.slice/crio-2bd8afcbe583b966d802360645b3f5ccfbb1740946eff0ef0f416d9f4d702df4 WatchSource:0}: Error finding container 2bd8afcbe583b966d802360645b3f5ccfbb1740946eff0ef0f416d9f4d702df4: Status 404 returned error can't find the container with id 2bd8afcbe583b966d802360645b3f5ccfbb1740946eff0ef0f416d9f4d702df4 Oct 30 21:25:32.328799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:32.328765 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j" event={"ID":"a99af3b4-3d20-4bb2-a59a-712fc18e35f8","Type":"ContainerStarted","Data":"a9da208783ebd08d7165b2bb7331a6902341908d170ab9904406a19b65fcdd2f"} Oct 30 21:25:32.329343 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:32.328817 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j" event={"ID":"a99af3b4-3d20-4bb2-a59a-712fc18e35f8","Type":"ContainerStarted","Data":"2bd8afcbe583b966d802360645b3f5ccfbb1740946eff0ef0f416d9f4d702df4"} Oct 30 21:25:32.370201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:32.370142 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j" podStartSLOduration=1.370122264 podStartE2EDuration="1.370122264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:25:32.368265312 +0000 UTC m=+1519.489339070" watchObservedRunningTime="2026-10-30 21:25:32.370122264 +0000 UTC m=+1519.491196032" Oct 30 21:25:33.338305 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:33.338270 4926 generic.go:356] "Generic (PLEG): container finished" podID="a99af3b4-3d20-4bb2-a59a-712fc18e35f8" containerID="a9da208783ebd08d7165b2bb7331a6902341908d170ab9904406a19b65fcdd2f" exitCode=0 Oct 30 21:25:33.338698 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:33.338355 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j" event={"ID":"a99af3b4-3d20-4bb2-a59a-712fc18e35f8","Type":"ContainerDied","Data":"a9da208783ebd08d7165b2bb7331a6902341908d170ab9904406a19b65fcdd2f"} Oct 30 21:25:35.008322 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:35.008289 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j" Oct 30 21:25:35.064498 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:35.064464 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a99af3b4-3d20-4bb2-a59a-712fc18e35f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99af3b4-3d20-4bb2-a59a-712fc18e35f8-maas-api-xks-ca\") pod \"a99af3b4-3d20-4bb2-a59a-712fc18e35f8\" (UID: \"a99af3b4-3d20-4bb2-a59a-712fc18e35f8\") " Oct 30 21:25:35.064649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:35.064593 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99af3b4-3d20-4bb2-a59a-712fc18e35f8-kube-api-access-c5hsp\" (UniqueName: \"kubernetes.io/projected/a99af3b4-3d20-4bb2-a59a-712fc18e35f8-kube-api-access-c5hsp\") pod \"a99af3b4-3d20-4bb2-a59a-712fc18e35f8\" (UID: \"a99af3b4-3d20-4bb2-a59a-712fc18e35f8\") " Oct 30 21:25:35.064922 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:35.064898 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a99af3b4-3d20-4bb2-a59a-712fc18e35f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99af3b4-3d20-4bb2-a59a-712fc18e35f8-maas-api-openshift-ca\") pod \"a99af3b4-3d20-4bb2-a59a-712fc18e35f8\" (UID: \"a99af3b4-3d20-4bb2-a59a-712fc18e35f8\") " Oct 30 21:25:35.066949 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:35.066913 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99af3b4-3d20-4bb2-a59a-712fc18e35f8-kube-api-access-c5hsp" pod "a99af3b4-3d20-4bb2-a59a-712fc18e35f8" (UID: "a99af3b4-3d20-4bb2-a59a-712fc18e35f8"). InnerVolumeSpecName "kube-api-access-c5hsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:25:35.067159 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:35.066981 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a99af3b4-3d20-4bb2-a59a-712fc18e35f8-maas-api-xks-ca" pod "a99af3b4-3d20-4bb2-a59a-712fc18e35f8" (UID: "a99af3b4-3d20-4bb2-a59a-712fc18e35f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:25:35.068908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:35.068883 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99af3b4-3d20-4bb2-a59a-712fc18e35f8-maas-api-openshift-ca" pod "a99af3b4-3d20-4bb2-a59a-712fc18e35f8" (UID: "a99af3b4-3d20-4bb2-a59a-712fc18e35f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:25:35.166795 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:35.166758 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5hsp\" (UniqueName: \"kubernetes.io/projected/a99af3b4-3d20-4bb2-a59a-712fc18e35f8-kube-api-access-c5hsp\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:25:35.166795 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:35.166795 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99af3b4-3d20-4bb2-a59a-712fc18e35f8-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:25:35.166991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:35.166810 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99af3b4-3d20-4bb2-a59a-712fc18e35f8-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:25:35.348757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:35.348603 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7"] Oct 30 21:25:35.353781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:35.353752 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-c1da2183-t87p7"] Oct 30 21:25:35.364813 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:35.364784 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j" Oct 30 21:25:35.364959 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:35.364819 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j" event={"ID":"a99af3b4-3d20-4bb2-a59a-712fc18e35f8","Type":"ContainerDied","Data":"2bd8afcbe583b966d802360645b3f5ccfbb1740946eff0ef0f416d9f4d702df4"} Oct 30 21:25:35.364959 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:35.364872 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bd8afcbe583b966d802360645b3f5ccfbb1740946eff0ef0f416d9f4d702df4" Oct 30 21:25:37.019325 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:37.019282 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="158962c8-8c25-4dc6-8c5f-31b330c15f96" path="/var/lib/kubelet/pods/158962c8-8c25-4dc6-8c5f-31b330c15f96/volumes" Oct 30 21:25:37.485395 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:25:37.485359 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:25:38.074104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:38.074019 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfae3730-snd5s"] Oct 30 21:25:38.075669 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:38.075442 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dfae3730-snd5s"] Oct 30 21:25:39.018374 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:39.018339 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4" path="/var/lib/kubelet/pods/d670edaa-7a1b-4f0c-83aa-0e6c2ae7daf4/volumes" Oct 30 21:25:59.703936 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:25:59.703895 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:26:00.862150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:00.862105 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4266b7-p94mm"] Oct 30 21:26:00.872601 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:00.872564 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4266b7-p94mm"] Oct 30 21:26:01.134116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:01.116954 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87545d7c-46fa-44bf-af98-3e94095693ff" path="/var/lib/kubelet/pods/87545d7c-46fa-44bf-af98-3e94095693ff/volumes" Oct 30 21:26:02.167344 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:02.167305 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w"] Oct 30 21:26:02.176952 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:02.176911 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2c7ca6-7c28w"] Oct 30 21:26:03.021865 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:03.021827 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8df84efc-a402-4766-b7ad-60de74be2e0e" path="/var/lib/kubelet/pods/8df84efc-a402-4766-b7ad-60de74be2e0e/volumes" Oct 30 21:26:07.488521 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:26:07.488476 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:26:11.407568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.407529 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r"] Oct 30 21:26:11.408637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.408609 4926 cpu_manager.go:414] "containerMap: removing container" podUID="a99af3b4-3d20-4bb2-a59a-712fc18e35f8" containerName="revoke-keys" Oct 30 21:26:11.408637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.408637 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a99af3b4-3d20-4bb2-a59a-712fc18e35f8" containerName="revoke-keys" Oct 30 21:26:11.408877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.408859 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="a99af3b4-3d20-4bb2-a59a-712fc18e35f8" containerName="revoke-keys" Oct 30 21:26:11.413615 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.413591 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.434414 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.434384 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c5145c-openshift-default-dockercfg-hm45t\"" Oct 30 21:26:11.443539 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.443507 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r"] Oct 30 21:26:11.519563 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.519526 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-envoy\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.519857 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.519840 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-workload-socket\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.519996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.519986 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-token\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.520190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.520174 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-workload-certs\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.520319 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.520306 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-credential-socket\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.520428 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.520419 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-podinfo\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.520569 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.520560 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk69z\" (UniqueName: \"kubernetes.io/projected/2eea6ad7-9d9b-4205-89d2-af24991331de-kube-api-access-tk69z\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.520632 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.520624 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-data\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.520717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.520708 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eea6ad7-9d9b-4205-89d2-af24991331de-istiod-ca-cert\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.624106 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.624064 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-token\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.641575 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.641539 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-workload-certs\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.642853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.642820 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-credential-socket\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.643178 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.643155 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-podinfo\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.643506 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.643483 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk69z\" (UniqueName: \"kubernetes.io/projected/2eea6ad7-9d9b-4205-89d2-af24991331de-kube-api-access-tk69z\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.643647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.643629 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-data\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.643775 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.643759 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eea6ad7-9d9b-4205-89d2-af24991331de-istiod-ca-cert\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.643991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.643969 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-envoy\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.644126 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.644107 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-data\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.644236 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.644175 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-workload-socket\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.644446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.644423 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-workload-socket\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.644531 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.643160 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-credential-socket\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.644531 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.642727 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-workload-certs\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.645567 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.645443 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eea6ad7-9d9b-4205-89d2-af24991331de-istiod-ca-cert\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.668753 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.668721 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-podinfo\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.679134 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.674745 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk69z\" (UniqueName: \"kubernetes.io/projected/2eea6ad7-9d9b-4205-89d2-af24991331de-kube-api-access-tk69z\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.679401 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.676223 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-token\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.692374 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.692336 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-envoy\") pod \"e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:11.765055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:11.760087 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:12.414520 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:12.414488 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r"] Oct 30 21:26:12.419177 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:26:12.419146 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eea6ad7_9d9b_4205_89d2_af24991331de.slice/crio-7deb4cb521eb927cb5f7d79a695ceebbf85d9e9d8e92fdcf6fc5fcd0657761bf WatchSource:0}: Error finding container 7deb4cb521eb927cb5f7d79a695ceebbf85d9e9d8e92fdcf6fc5fcd0657761bf: Status 404 returned error can't find the container with id 7deb4cb521eb927cb5f7d79a695ceebbf85d9e9d8e92fdcf6fc5fcd0657761bf Oct 30 21:26:12.422369 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:12.422335 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:26:12.422478 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:12.422408 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:26:12.422478 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:12.422440 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:26:12.872977 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:12.872924 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" event={"ID":"2eea6ad7-9d9b-4205-89d2-af24991331de","Type":"ContainerStarted","Data":"d43d73aee50bbddfb466a095c47aab5d3f4afef733d0c6c96e6b2c39dc15637d"} Oct 30 21:26:12.873210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:12.872994 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" event={"ID":"2eea6ad7-9d9b-4205-89d2-af24991331de","Type":"ContainerStarted","Data":"7deb4cb521eb927cb5f7d79a695ceebbf85d9e9d8e92fdcf6fc5fcd0657761bf"} Oct 30 21:26:12.942599 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:12.942535 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" podStartSLOduration=1.9425185329999999 podStartE2EDuration="1.942518533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:12.926725329 +0000 UTC m=+1560.047799076" watchObservedRunningTime="2026-10-30 21:26:12.942518533 +0000 UTC m=+1560.063592347" Oct 30 21:26:13.760849 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:13.760815 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:13.772393 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:13.772360 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:13.884873 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:13.884837 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:13.886181 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:13.886155 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:26:27.039857 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:27.039760 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv"] Oct 30 21:26:27.040737 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:27.040679 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" podUID="1ce21589-5994-4cd8-b978-450027ab13ec" containerName="istio-proxy" containerID="cri-o://53f2daf36737829dddb318b004c58df08401cfa4ba55c098c16dabdfa52ee220" gracePeriod=30 Oct 30 21:26:32.344093 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.343607 4926 generic.go:356] "Generic (PLEG): container finished" podID="1ce21589-5994-4cd8-b978-450027ab13ec" containerID="53f2daf36737829dddb318b004c58df08401cfa4ba55c098c16dabdfa52ee220" exitCode=0 Oct 30 21:26:32.344093 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.343920 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" event={"ID":"1ce21589-5994-4cd8-b978-450027ab13ec","Type":"ContainerDied","Data":"53f2daf36737829dddb318b004c58df08401cfa4ba55c098c16dabdfa52ee220"} Oct 30 21:26:32.439718 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.439688 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:26:32.515770 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.515717 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-istio-envoy\") pod \"1ce21589-5994-4cd8-b978-450027ab13ec\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " Oct 30 21:26:32.516093 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.516068 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce21589-5994-4cd8-b978-450027ab13ec-kube-api-access-n5rgl\" (UniqueName: \"kubernetes.io/projected/1ce21589-5994-4cd8-b978-450027ab13ec-kube-api-access-n5rgl\") pod \"1ce21589-5994-4cd8-b978-450027ab13ec\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " Oct 30 21:26:32.516267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.516239 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ce21589-5994-4cd8-b978-450027ab13ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce21589-5994-4cd8-b978-450027ab13ec-istiod-ca-cert\") pod \"1ce21589-5994-4cd8-b978-450027ab13ec\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " Oct 30 21:26:32.516394 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.516288 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-istio-data\") pod \"1ce21589-5994-4cd8-b978-450027ab13ec\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " Oct 30 21:26:32.516453 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.516402 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ce21589-5994-4cd8-b978-450027ab13ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce21589-5994-4cd8-b978-450027ab13ec-istio-podinfo\") pod \"1ce21589-5994-4cd8-b978-450027ab13ec\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " Oct 30 21:26:32.516504 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.516476 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-workload-certs\") pod \"1ce21589-5994-4cd8-b978-450027ab13ec\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " Oct 30 21:26:32.516740 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.516720 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-credential-socket\") pod \"1ce21589-5994-4cd8-b978-450027ab13ec\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " Oct 30 21:26:32.516897 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.516870 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-workload-socket\") pod \"1ce21589-5994-4cd8-b978-450027ab13ec\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " Oct 30 21:26:32.516958 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.516942 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ce21589-5994-4cd8-b978-450027ab13ec-istio-token\" (UniqueName: \"kubernetes.io/projected/1ce21589-5994-4cd8-b978-450027ab13ec-istio-token\") pod \"1ce21589-5994-4cd8-b978-450027ab13ec\" (UID: \"1ce21589-5994-4cd8-b978-450027ab13ec\") " Oct 30 21:26:32.521150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.521115 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-istio-data" pod "1ce21589-5994-4cd8-b978-450027ab13ec" (UID: "1ce21589-5994-4cd8-b978-450027ab13ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:32.521961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.521921 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce21589-5994-4cd8-b978-450027ab13ec-istio-token" pod "1ce21589-5994-4cd8-b978-450027ab13ec" (UID: "1ce21589-5994-4cd8-b978-450027ab13ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:32.524357 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.524187 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-credential-socket" pod "1ce21589-5994-4cd8-b978-450027ab13ec" (UID: "1ce21589-5994-4cd8-b978-450027ab13ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:32.524464 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.524392 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ce21589-5994-4cd8-b978-450027ab13ec-istio-podinfo" pod "1ce21589-5994-4cd8-b978-450027ab13ec" (UID: "1ce21589-5994-4cd8-b978-450027ab13ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:26:32.524812 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.524576 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-workload-certs" pod "1ce21589-5994-4cd8-b978-450027ab13ec" (UID: "1ce21589-5994-4cd8-b978-450027ab13ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:32.524812 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.524599 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-workload-socket" pod "1ce21589-5994-4cd8-b978-450027ab13ec" (UID: "1ce21589-5994-4cd8-b978-450027ab13ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:32.524952 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.524926 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ce21589-5994-4cd8-b978-450027ab13ec-istiod-ca-cert" pod "1ce21589-5994-4cd8-b978-450027ab13ec" (UID: "1ce21589-5994-4cd8-b978-450027ab13ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:26:32.538642 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.538609 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-istio-envoy" pod "1ce21589-5994-4cd8-b978-450027ab13ec" (UID: "1ce21589-5994-4cd8-b978-450027ab13ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:32.540539 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.540509 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce21589-5994-4cd8-b978-450027ab13ec-kube-api-access-n5rgl" pod "1ce21589-5994-4cd8-b978-450027ab13ec" (UID: "1ce21589-5994-4cd8-b978-450027ab13ec"). InnerVolumeSpecName "kube-api-access-n5rgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:32.619970 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.619878 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:32.619970 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.619920 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ce21589-5994-4cd8-b978-450027ab13ec-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:32.619970 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.619934 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:32.619970 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.619947 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5rgl\" (UniqueName: \"kubernetes.io/projected/1ce21589-5994-4cd8-b978-450027ab13ec-kube-api-access-n5rgl\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:32.619970 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.619963 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ce21589-5994-4cd8-b978-450027ab13ec-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:32.620271 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.619977 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:32.620271 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.619990 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ce21589-5994-4cd8-b978-450027ab13ec-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:32.620271 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.620003 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:32.620271 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:32.620015 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ce21589-5994-4cd8-b978-450027ab13ec-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:33.353627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:33.353535 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" event={"ID":"1ce21589-5994-4cd8-b978-450027ab13ec","Type":"ContainerDied","Data":"159acd8e35b97fb2f0a417e8e7ee434e57f9cd615f844e32f21808c4d4ee10af"} Oct 30 21:26:33.353627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:33.353566 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv" Oct 30 21:26:33.353627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:33.353611 4926 scope.go:122] "RemoveContainer" containerID="53f2daf36737829dddb318b004c58df08401cfa4ba55c098c16dabdfa52ee220" Oct 30 21:26:33.422689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:33.422654 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv"] Oct 30 21:26:33.434227 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:33.434198 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-c1da2183-openshift-default-746fc598bf-gghdv"] Oct 30 21:26:33.829530 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:33.829494 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cee0a8d7-qbghx"] Oct 30 21:26:33.830248 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:33.830228 4926 cpu_manager.go:414] "containerMap: removing container" podUID="1ce21589-5994-4cd8-b978-450027ab13ec" containerName="istio-proxy" Oct 30 21:26:33.830366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:33.830248 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ce21589-5994-4cd8-b978-450027ab13ec" containerName="istio-proxy" Oct 30 21:26:33.830484 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:33.830470 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ce21589-5994-4cd8-b978-450027ab13ec" containerName="istio-proxy" Oct 30 21:26:33.835611 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:33.835586 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee0a8d7-qbghx" Oct 30 21:26:33.853640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:33.853610 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:26:33.855203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:33.855177 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:26:33.856309 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:33.856288 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cee0a8d7-qbghx"] Oct 30 21:26:33.942841 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:33.942808 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a804628a-e494-4335-9871-5e317b38c9dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cee0a8d7-qbghx\" (UID: \"a804628a-e494-4335-9871-5e317b38c9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee0a8d7-qbghx" Oct 30 21:26:33.943077 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:33.942851 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a804628a-e494-4335-9871-5e317b38c9dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cee0a8d7-qbghx\" (UID: \"a804628a-e494-4335-9871-5e317b38c9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee0a8d7-qbghx" Oct 30 21:26:33.943132 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:33.943086 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8pbz\" (UniqueName: \"kubernetes.io/projected/a804628a-e494-4335-9871-5e317b38c9dd-kube-api-access-j8pbz\") pod \"maas-api-revoke-sub-cee0a8d7-qbghx\" (UID: \"a804628a-e494-4335-9871-5e317b38c9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee0a8d7-qbghx" Oct 30 21:26:34.045170 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:34.045138 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a804628a-e494-4335-9871-5e317b38c9dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cee0a8d7-qbghx\" (UID: \"a804628a-e494-4335-9871-5e317b38c9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee0a8d7-qbghx" Oct 30 21:26:34.045380 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:34.045180 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a804628a-e494-4335-9871-5e317b38c9dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cee0a8d7-qbghx\" (UID: \"a804628a-e494-4335-9871-5e317b38c9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee0a8d7-qbghx" Oct 30 21:26:34.045380 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:34.045242 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8pbz\" (UniqueName: \"kubernetes.io/projected/a804628a-e494-4335-9871-5e317b38c9dd-kube-api-access-j8pbz\") pod \"maas-api-revoke-sub-cee0a8d7-qbghx\" (UID: \"a804628a-e494-4335-9871-5e317b38c9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee0a8d7-qbghx" Oct 30 21:26:34.045882 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:34.045847 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a804628a-e494-4335-9871-5e317b38c9dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cee0a8d7-qbghx\" (UID: \"a804628a-e494-4335-9871-5e317b38c9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee0a8d7-qbghx" Oct 30 21:26:34.047723 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:34.047693 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a804628a-e494-4335-9871-5e317b38c9dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cee0a8d7-qbghx\" (UID: \"a804628a-e494-4335-9871-5e317b38c9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee0a8d7-qbghx" Oct 30 21:26:34.072095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:34.068664 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8pbz\" (UniqueName: \"kubernetes.io/projected/a804628a-e494-4335-9871-5e317b38c9dd-kube-api-access-j8pbz\") pod \"maas-api-revoke-sub-cee0a8d7-qbghx\" (UID: \"a804628a-e494-4335-9871-5e317b38c9dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee0a8d7-qbghx" Oct 30 21:26:34.150195 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:34.150116 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee0a8d7-qbghx" Oct 30 21:26:34.603203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:34.603165 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cee0a8d7-qbghx"] Oct 30 21:26:35.018214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:35.018180 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ce21589-5994-4cd8-b978-450027ab13ec" path="/var/lib/kubelet/pods/1ce21589-5994-4cd8-b978-450027ab13ec/volumes" Oct 30 21:26:35.614850 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:35.614808 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee0a8d7-qbghx" event={"ID":"a804628a-e494-4335-9871-5e317b38c9dd","Type":"ContainerStarted","Data":"f298dda00e2d18651a858ffa0a6f7e9b6545309e9aba8421b24a9745189933c7"} Oct 30 21:26:35.615260 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:35.614880 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee0a8d7-qbghx" event={"ID":"a804628a-e494-4335-9871-5e317b38c9dd","Type":"ContainerStarted","Data":"de572f1b30aceb266a01afd542c9055e8a1354245535fc2d75f93d2c31f1c766"} Oct 30 21:26:36.658739 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:36.658298 4926 generic.go:356] "Generic (PLEG): container finished" podID="a804628a-e494-4335-9871-5e317b38c9dd" containerID="f298dda00e2d18651a858ffa0a6f7e9b6545309e9aba8421b24a9745189933c7" exitCode=0 Oct 30 21:26:36.658739 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:36.658495 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee0a8d7-qbghx" event={"ID":"a804628a-e494-4335-9871-5e317b38c9dd","Type":"ContainerDied","Data":"f298dda00e2d18651a858ffa0a6f7e9b6545309e9aba8421b24a9745189933c7"} Oct 30 21:26:37.087678 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:37.086426 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j"] Oct 30 21:26:37.087678 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:37.087088 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee0a8d7-qbghx" Oct 30 21:26:37.091096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:37.090983 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ca18b3-9xh8j"] Oct 30 21:26:37.252444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:37.252353 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a804628a-e494-4335-9871-5e317b38c9dd-kube-api-access-j8pbz\" (UniqueName: \"kubernetes.io/projected/a804628a-e494-4335-9871-5e317b38c9dd-kube-api-access-j8pbz\") pod \"a804628a-e494-4335-9871-5e317b38c9dd\" (UID: \"a804628a-e494-4335-9871-5e317b38c9dd\") " Oct 30 21:26:37.252629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:37.252491 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a804628a-e494-4335-9871-5e317b38c9dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a804628a-e494-4335-9871-5e317b38c9dd-maas-api-xks-ca\") pod \"a804628a-e494-4335-9871-5e317b38c9dd\" (UID: \"a804628a-e494-4335-9871-5e317b38c9dd\") " Oct 30 21:26:37.252774 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:37.252746 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a804628a-e494-4335-9871-5e317b38c9dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a804628a-e494-4335-9871-5e317b38c9dd-maas-api-openshift-ca\") pod \"a804628a-e494-4335-9871-5e317b38c9dd\" (UID: \"a804628a-e494-4335-9871-5e317b38c9dd\") " Oct 30 21:26:37.253157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:37.253131 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a804628a-e494-4335-9871-5e317b38c9dd-maas-api-openshift-ca" pod "a804628a-e494-4335-9871-5e317b38c9dd" (UID: "a804628a-e494-4335-9871-5e317b38c9dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:26:37.253477 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:37.253460 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a804628a-e494-4335-9871-5e317b38c9dd-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:37.254623 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:37.254588 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a804628a-e494-4335-9871-5e317b38c9dd-maas-api-xks-ca" pod "a804628a-e494-4335-9871-5e317b38c9dd" (UID: "a804628a-e494-4335-9871-5e317b38c9dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:26:37.254718 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:37.254680 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a804628a-e494-4335-9871-5e317b38c9dd-kube-api-access-j8pbz" pod "a804628a-e494-4335-9871-5e317b38c9dd" (UID: "a804628a-e494-4335-9871-5e317b38c9dd"). InnerVolumeSpecName "kube-api-access-j8pbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:37.355273 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:37.355239 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8pbz\" (UniqueName: \"kubernetes.io/projected/a804628a-e494-4335-9871-5e317b38c9dd-kube-api-access-j8pbz\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:37.355273 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:37.355270 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a804628a-e494-4335-9871-5e317b38c9dd-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:37.491455 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:26:37.491406 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:26:37.668085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:37.668051 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee0a8d7-qbghx" Oct 30 21:26:37.668085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:37.668063 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cee0a8d7-qbghx" event={"ID":"a804628a-e494-4335-9871-5e317b38c9dd","Type":"ContainerDied","Data":"de572f1b30aceb266a01afd542c9055e8a1354245535fc2d75f93d2c31f1c766"} Oct 30 21:26:37.668545 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:37.668128 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de572f1b30aceb266a01afd542c9055e8a1354245535fc2d75f93d2c31f1c766" Oct 30 21:26:39.026392 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:39.026357 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a99af3b4-3d20-4bb2-a59a-712fc18e35f8" path="/var/lib/kubelet/pods/a99af3b4-3d20-4bb2-a59a-712fc18e35f8/volumes" Oct 30 21:26:46.613945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:46.613884 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z"] Oct 30 21:26:46.614600 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:46.614374 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" podUID="226ee133-aef2-4cd2-b0bb-5bb745a03862" containerName="istio-proxy" containerID="cri-o://05cfd4fb21377a75dd0b48285493154c297d94a6421f5f7f84c4dd3411b75e14" gracePeriod=30 Oct 30 21:26:46.770141 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:46.770095 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk"] Oct 30 21:26:46.770679 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:46.770618 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" podUID="0e6eca6b-e95a-44c5-bc67-f431a266729d" containerName="istio-proxy" containerID="cri-o://7f2ace4c3b3c01f884d885dee065419c64fe53d61c010c2fb2e8b40a4eff8cc7" gracePeriod=30 Oct 30 21:26:47.264740 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:47.264685 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf"] Oct 30 21:26:47.287069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:47.266018 4926 cpu_manager.go:414] "containerMap: removing container" podUID="a804628a-e494-4335-9871-5e317b38c9dd" containerName="revoke-keys" Oct 30 21:26:47.287309 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:47.287286 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a804628a-e494-4335-9871-5e317b38c9dd" containerName="revoke-keys" Oct 30 21:26:47.287716 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:47.287701 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="a804628a-e494-4335-9871-5e317b38c9dd" containerName="revoke-keys" Oct 30 21:26:47.330078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:47.326315 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf"] Oct 30 21:26:47.330078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:47.326479 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf" Oct 30 21:26:47.382985 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:47.371353 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:26:47.382985 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:47.371443 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:26:47.505382 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:47.505276 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq659\" (UniqueName: \"kubernetes.io/projected/b30f1ce4-1915-46fa-8393-9e3ffbf922ea-kube-api-access-hq659\") pod \"maas-api-revoke-keys-e2e-api-b-517366-x4mlf\" (UID: \"b30f1ce4-1915-46fa-8393-9e3ffbf922ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf" Oct 30 21:26:47.505591 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:47.505403 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b30f1ce4-1915-46fa-8393-9e3ffbf922ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-517366-x4mlf\" (UID: \"b30f1ce4-1915-46fa-8393-9e3ffbf922ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf" Oct 30 21:26:47.505591 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:47.505511 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b30f1ce4-1915-46fa-8393-9e3ffbf922ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-517366-x4mlf\" (UID: \"b30f1ce4-1915-46fa-8393-9e3ffbf922ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf" Oct 30 21:26:47.607417 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:47.607319 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq659\" (UniqueName: \"kubernetes.io/projected/b30f1ce4-1915-46fa-8393-9e3ffbf922ea-kube-api-access-hq659\") pod \"maas-api-revoke-keys-e2e-api-b-517366-x4mlf\" (UID: \"b30f1ce4-1915-46fa-8393-9e3ffbf922ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf" Oct 30 21:26:47.608210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:47.608180 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b30f1ce4-1915-46fa-8393-9e3ffbf922ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-517366-x4mlf\" (UID: \"b30f1ce4-1915-46fa-8393-9e3ffbf922ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf" Oct 30 21:26:47.609503 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:47.609482 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b30f1ce4-1915-46fa-8393-9e3ffbf922ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-517366-x4mlf\" (UID: \"b30f1ce4-1915-46fa-8393-9e3ffbf922ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf" Oct 30 21:26:47.610141 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:47.609271 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b30f1ce4-1915-46fa-8393-9e3ffbf922ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-517366-x4mlf\" (UID: \"b30f1ce4-1915-46fa-8393-9e3ffbf922ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf" Oct 30 21:26:47.613572 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:47.613544 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b30f1ce4-1915-46fa-8393-9e3ffbf922ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-517366-x4mlf\" (UID: \"b30f1ce4-1915-46fa-8393-9e3ffbf922ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf" Oct 30 21:26:47.627998 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:47.627966 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq659\" (UniqueName: \"kubernetes.io/projected/b30f1ce4-1915-46fa-8393-9e3ffbf922ea-kube-api-access-hq659\") pod \"maas-api-revoke-keys-e2e-api-b-517366-x4mlf\" (UID: \"b30f1ce4-1915-46fa-8393-9e3ffbf922ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf" Oct 30 21:26:47.660851 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:47.660812 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf" Oct 30 21:26:48.055102 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:48.052156 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf"] Oct 30 21:26:48.055102 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:48.052374 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" podUID="472a982c-6acb-4164-96f6-5cf2904a4c8d" containerName="istio-proxy" containerID="cri-o://5a9de4e8e86b80d8baaebb07cf93d204ba0ce046332869c8352c24fa936029fb" gracePeriod=30 Oct 30 21:26:48.065524 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:48.058773 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj"] Oct 30 21:26:48.065524 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:48.059010 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" podUID="4a6c48f2-e476-4ae9-82b4-b476a225350d" containerName="istio-proxy" containerID="cri-o://8c9373da8610358a4fd85e0238b3633925c6510a2798c31c497d3bee985b870d" gracePeriod=30 Oct 30 21:26:48.321162 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:48.321110 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf"] Oct 30 21:26:49.217055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:49.216435 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf" event={"ID":"b30f1ce4-1915-46fa-8393-9e3ffbf922ea","Type":"ContainerStarted","Data":"57d3e66db12997850149ebc1afadf57788a12f8e068ad9985fb0a603d08fe1a3"} Oct 30 21:26:49.217055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:49.216553 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf" event={"ID":"b30f1ce4-1915-46fa-8393-9e3ffbf922ea","Type":"ContainerStarted","Data":"db3d7b2aa5778d01364d4647ed338587ff6316d067e52fc22f8f7f2363226998"} Oct 30 21:26:49.273831 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:49.273736 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf" podStartSLOduration=2.27372011 podStartE2EDuration="2.27372011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:49.269064732 +0000 UTC m=+1596.390138473" watchObservedRunningTime="2026-10-30 21:26:49.27372011 +0000 UTC m=+1596.394793862" Oct 30 21:26:50.229475 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:50.229387 4926 generic.go:356] "Generic (PLEG): container finished" podID="b30f1ce4-1915-46fa-8393-9e3ffbf922ea" containerID="57d3e66db12997850149ebc1afadf57788a12f8e068ad9985fb0a603d08fe1a3" exitCode=0 Oct 30 21:26:50.229475 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:50.229437 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf" event={"ID":"b30f1ce4-1915-46fa-8393-9e3ffbf922ea","Type":"ContainerDied","Data":"57d3e66db12997850149ebc1afadf57788a12f8e068ad9985fb0a603d08fe1a3"} Oct 30 21:26:51.481208 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:51.481174 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf" Oct 30 21:26:51.534641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:51.534549 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b30f1ce4-1915-46fa-8393-9e3ffbf922ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b30f1ce4-1915-46fa-8393-9e3ffbf922ea-maas-api-xks-ca\") pod \"b30f1ce4-1915-46fa-8393-9e3ffbf922ea\" (UID: \"b30f1ce4-1915-46fa-8393-9e3ffbf922ea\") " Oct 30 21:26:51.534641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:51.534632 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30f1ce4-1915-46fa-8393-9e3ffbf922ea-kube-api-access-hq659\" (UniqueName: \"kubernetes.io/projected/b30f1ce4-1915-46fa-8393-9e3ffbf922ea-kube-api-access-hq659\") pod \"b30f1ce4-1915-46fa-8393-9e3ffbf922ea\" (UID: \"b30f1ce4-1915-46fa-8393-9e3ffbf922ea\") " Oct 30 21:26:51.535115 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:51.534976 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b30f1ce4-1915-46fa-8393-9e3ffbf922ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b30f1ce4-1915-46fa-8393-9e3ffbf922ea-maas-api-openshift-ca\") pod \"b30f1ce4-1915-46fa-8393-9e3ffbf922ea\" (UID: \"b30f1ce4-1915-46fa-8393-9e3ffbf922ea\") " Oct 30 21:26:51.535425 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:51.535392 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b30f1ce4-1915-46fa-8393-9e3ffbf922ea-maas-api-openshift-ca" pod "b30f1ce4-1915-46fa-8393-9e3ffbf922ea" (UID: "b30f1ce4-1915-46fa-8393-9e3ffbf922ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:26:51.535630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:51.535603 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b30f1ce4-1915-46fa-8393-9e3ffbf922ea-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:51.537606 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:51.537580 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b30f1ce4-1915-46fa-8393-9e3ffbf922ea-maas-api-xks-ca" pod "b30f1ce4-1915-46fa-8393-9e3ffbf922ea" (UID: "b30f1ce4-1915-46fa-8393-9e3ffbf922ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:26:51.539797 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:51.539769 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30f1ce4-1915-46fa-8393-9e3ffbf922ea-kube-api-access-hq659" pod "b30f1ce4-1915-46fa-8393-9e3ffbf922ea" (UID: "b30f1ce4-1915-46fa-8393-9e3ffbf922ea"). InnerVolumeSpecName "kube-api-access-hq659". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:51.637922 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:51.637882 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b30f1ce4-1915-46fa-8393-9e3ffbf922ea-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:51.637922 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:51.637922 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq659\" (UniqueName: \"kubernetes.io/projected/b30f1ce4-1915-46fa-8393-9e3ffbf922ea-kube-api-access-hq659\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:51.987291 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:51.987257 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:26:52.046099 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.046000 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/226ee133-aef2-4cd2-b0bb-5bb745a03862-kube-api-access-nbfc5\" (UniqueName: \"kubernetes.io/projected/226ee133-aef2-4cd2-b0bb-5bb745a03862-kube-api-access-nbfc5\") pod \"226ee133-aef2-4cd2-b0bb-5bb745a03862\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " Oct 30 21:26:52.046307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.046125 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-token\" (UniqueName: \"kubernetes.io/projected/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-token\") pod \"226ee133-aef2-4cd2-b0bb-5bb745a03862\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " Oct 30 21:26:52.046307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.046244 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-data\") pod \"226ee133-aef2-4cd2-b0bb-5bb745a03862\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " Oct 30 21:26:52.046459 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.046439 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-envoy\") pod \"226ee133-aef2-4cd2-b0bb-5bb745a03862\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " Oct 30 21:26:52.046525 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.046516 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/226ee133-aef2-4cd2-b0bb-5bb745a03862-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/226ee133-aef2-4cd2-b0bb-5bb745a03862-istiod-ca-cert\") pod \"226ee133-aef2-4cd2-b0bb-5bb745a03862\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " Oct 30 21:26:52.046584 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.046545 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-podinfo\") pod \"226ee133-aef2-4cd2-b0bb-5bb745a03862\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " Oct 30 21:26:52.046645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.046596 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-workload-certs\") pod \"226ee133-aef2-4cd2-b0bb-5bb745a03862\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " Oct 30 21:26:52.046645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.046629 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-credential-socket\") pod \"226ee133-aef2-4cd2-b0bb-5bb745a03862\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " Oct 30 21:26:52.046801 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.046669 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-workload-socket\") pod \"226ee133-aef2-4cd2-b0bb-5bb745a03862\" (UID: \"226ee133-aef2-4cd2-b0bb-5bb745a03862\") " Oct 30 21:26:52.050245 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.050209 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-credential-socket" pod "226ee133-aef2-4cd2-b0bb-5bb745a03862" (UID: "226ee133-aef2-4cd2-b0bb-5bb745a03862"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:52.050475 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.050273 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-workload-certs" pod "226ee133-aef2-4cd2-b0bb-5bb745a03862" (UID: "226ee133-aef2-4cd2-b0bb-5bb745a03862"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:52.050475 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.050446 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-workload-socket" pod "226ee133-aef2-4cd2-b0bb-5bb745a03862" (UID: "226ee133-aef2-4cd2-b0bb-5bb745a03862"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:52.051010 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.050864 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-data" pod "226ee133-aef2-4cd2-b0bb-5bb745a03862" (UID: "226ee133-aef2-4cd2-b0bb-5bb745a03862"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:52.053877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.053848 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/226ee133-aef2-4cd2-b0bb-5bb745a03862-istiod-ca-cert" pod "226ee133-aef2-4cd2-b0bb-5bb745a03862" (UID: "226ee133-aef2-4cd2-b0bb-5bb745a03862"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:26:52.055057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.055016 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-token" pod "226ee133-aef2-4cd2-b0bb-5bb745a03862" (UID: "226ee133-aef2-4cd2-b0bb-5bb745a03862"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:52.055186 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.055162 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-podinfo" pod "226ee133-aef2-4cd2-b0bb-5bb745a03862" (UID: "226ee133-aef2-4cd2-b0bb-5bb745a03862"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:26:52.057011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.056987 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-envoy" pod "226ee133-aef2-4cd2-b0bb-5bb745a03862" (UID: "226ee133-aef2-4cd2-b0bb-5bb745a03862"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:52.057254 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.057228 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226ee133-aef2-4cd2-b0bb-5bb745a03862-kube-api-access-nbfc5" pod "226ee133-aef2-4cd2-b0bb-5bb745a03862" (UID: "226ee133-aef2-4cd2-b0bb-5bb745a03862"). InnerVolumeSpecName "kube-api-access-nbfc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:52.149470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.149437 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:52.149470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.149474 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/226ee133-aef2-4cd2-b0bb-5bb745a03862-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:52.149660 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.149490 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:52.149660 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.149503 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:52.149660 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.149516 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:52.149660 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.149531 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:52.149660 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.149544 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbfc5\" (UniqueName: \"kubernetes.io/projected/226ee133-aef2-4cd2-b0bb-5bb745a03862-kube-api-access-nbfc5\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:52.149660 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.149556 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:52.149660 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.149569 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/226ee133-aef2-4cd2-b0bb-5bb745a03862-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:52.150452 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.150428 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:26:52.250657 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.250623 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-credential-socket\") pod \"0e6eca6b-e95a-44c5-bc67-f431a266729d\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " Oct 30 21:26:52.250877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.250692 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-workload-socket\") pod \"0e6eca6b-e95a-44c5-bc67-f431a266729d\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " Oct 30 21:26:52.250877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.250763 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-podinfo\") pod \"0e6eca6b-e95a-44c5-bc67-f431a266729d\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " Oct 30 21:26:52.250877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.250815 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e6eca6b-e95a-44c5-bc67-f431a266729d-kube-api-access-rtvfd\" (UniqueName: \"kubernetes.io/projected/0e6eca6b-e95a-44c5-bc67-f431a266729d-kube-api-access-rtvfd\") pod \"0e6eca6b-e95a-44c5-bc67-f431a266729d\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " Oct 30 21:26:52.251069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.250890 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-data\") pod \"0e6eca6b-e95a-44c5-bc67-f431a266729d\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " Oct 30 21:26:52.251069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.250963 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-token\" (UniqueName: \"kubernetes.io/projected/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-token\") pod \"0e6eca6b-e95a-44c5-bc67-f431a266729d\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " Oct 30 21:26:52.251069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.250996 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-credential-socket" pod "0e6eca6b-e95a-44c5-bc67-f431a266729d" (UID: "0e6eca6b-e95a-44c5-bc67-f431a266729d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:52.251069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.251017 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-workload-certs\") pod \"0e6eca6b-e95a-44c5-bc67-f431a266729d\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " Oct 30 21:26:52.251289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.251092 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-envoy\") pod \"0e6eca6b-e95a-44c5-bc67-f431a266729d\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " Oct 30 21:26:52.251289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.251153 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e6eca6b-e95a-44c5-bc67-f431a266729d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6eca6b-e95a-44c5-bc67-f431a266729d-istiod-ca-cert\") pod \"0e6eca6b-e95a-44c5-bc67-f431a266729d\" (UID: \"0e6eca6b-e95a-44c5-bc67-f431a266729d\") " Oct 30 21:26:52.251776 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.251745 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-workload-socket" pod "0e6eca6b-e95a-44c5-bc67-f431a266729d" (UID: "0e6eca6b-e95a-44c5-bc67-f431a266729d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:52.252830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.252795 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-data" pod "0e6eca6b-e95a-44c5-bc67-f431a266729d" (UID: "0e6eca6b-e95a-44c5-bc67-f431a266729d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:52.252935 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.252812 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:52.252935 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.252862 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:52.253069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.253011 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e6eca6b-e95a-44c5-bc67-f431a266729d-istiod-ca-cert" pod "0e6eca6b-e95a-44c5-bc67-f431a266729d" (UID: "0e6eca6b-e95a-44c5-bc67-f431a266729d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:26:52.255316 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.255277 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-envoy" pod "0e6eca6b-e95a-44c5-bc67-f431a266729d" (UID: "0e6eca6b-e95a-44c5-bc67-f431a266729d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:52.256295 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.256239 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-workload-certs" pod "0e6eca6b-e95a-44c5-bc67-f431a266729d" (UID: "0e6eca6b-e95a-44c5-bc67-f431a266729d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:52.258082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.257866 4926 generic.go:356] "Generic (PLEG): container finished" podID="0e6eca6b-e95a-44c5-bc67-f431a266729d" containerID="7f2ace4c3b3c01f884d885dee065419c64fe53d61c010c2fb2e8b40a4eff8cc7" exitCode=0 Oct 30 21:26:52.258183 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.258118 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" event={"ID":"0e6eca6b-e95a-44c5-bc67-f431a266729d","Type":"ContainerDied","Data":"7f2ace4c3b3c01f884d885dee065419c64fe53d61c010c2fb2e8b40a4eff8cc7"} Oct 30 21:26:52.258183 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.258175 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" event={"ID":"0e6eca6b-e95a-44c5-bc67-f431a266729d","Type":"ContainerDied","Data":"314beda9e5a2d1625d292bf1a3c239810d3494cf51714696194d2a5ee4e4b800"} Oct 30 21:26:52.258310 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.258192 4926 scope.go:122] "RemoveContainer" containerID="7f2ace4c3b3c01f884d885dee065419c64fe53d61c010c2fb2e8b40a4eff8cc7" Oct 30 21:26:52.258370 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.258180 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk" Oct 30 21:26:52.261239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.261216 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-podinfo" pod "0e6eca6b-e95a-44c5-bc67-f431a266729d" (UID: "0e6eca6b-e95a-44c5-bc67-f431a266729d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:26:52.261770 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.261743 4926 generic.go:356] "Generic (PLEG): container finished" podID="226ee133-aef2-4cd2-b0bb-5bb745a03862" containerID="05cfd4fb21377a75dd0b48285493154c297d94a6421f5f7f84c4dd3411b75e14" exitCode=0 Oct 30 21:26:52.261859 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.261807 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" Oct 30 21:26:52.261930 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.261854 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" event={"ID":"226ee133-aef2-4cd2-b0bb-5bb745a03862","Type":"ContainerDied","Data":"05cfd4fb21377a75dd0b48285493154c297d94a6421f5f7f84c4dd3411b75e14"} Oct 30 21:26:52.261930 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.261921 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z" event={"ID":"226ee133-aef2-4cd2-b0bb-5bb745a03862","Type":"ContainerDied","Data":"33dbea3bb03ae53123c1b7db31babc5399ab5926f56e8fba7e0d16726469660b"} Oct 30 21:26:52.263730 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.263677 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf" Oct 30 21:26:52.263730 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.263719 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf" event={"ID":"b30f1ce4-1915-46fa-8393-9e3ffbf922ea","Type":"ContainerDied","Data":"db3d7b2aa5778d01364d4647ed338587ff6316d067e52fc22f8f7f2363226998"} Oct 30 21:26:52.263877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.263757 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6eca6b-e95a-44c5-bc67-f431a266729d-kube-api-access-rtvfd" pod "0e6eca6b-e95a-44c5-bc67-f431a266729d" (UID: "0e6eca6b-e95a-44c5-bc67-f431a266729d"). InnerVolumeSpecName "kube-api-access-rtvfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:52.263877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.263766 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db3d7b2aa5778d01364d4647ed338587ff6316d067e52fc22f8f7f2363226998" Oct 30 21:26:52.265470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.265431 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-token" pod "0e6eca6b-e95a-44c5-bc67-f431a266729d" (UID: "0e6eca6b-e95a-44c5-bc67-f431a266729d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:52.297498 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.297467 4926 scope.go:122] "RemoveContainer" containerID="7f2ace4c3b3c01f884d885dee065419c64fe53d61c010c2fb2e8b40a4eff8cc7" Oct 30 21:26:52.299247 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:26:52.299094 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f2ace4c3b3c01f884d885dee065419c64fe53d61c010c2fb2e8b40a4eff8cc7\": container with ID starting with 7f2ace4c3b3c01f884d885dee065419c64fe53d61c010c2fb2e8b40a4eff8cc7 not found: ID does not exist" containerID="7f2ace4c3b3c01f884d885dee065419c64fe53d61c010c2fb2e8b40a4eff8cc7" Oct 30 21:26:52.299247 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.299135 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f2ace4c3b3c01f884d885dee065419c64fe53d61c010c2fb2e8b40a4eff8cc7"} err="failed to get container status \"7f2ace4c3b3c01f884d885dee065419c64fe53d61c010c2fb2e8b40a4eff8cc7\": rpc error: code = NotFound desc = could not find container \"7f2ace4c3b3c01f884d885dee065419c64fe53d61c010c2fb2e8b40a4eff8cc7\": container with ID starting with 7f2ace4c3b3c01f884d885dee065419c64fe53d61c010c2fb2e8b40a4eff8cc7 not found: ID does not exist" Oct 30 21:26:52.299247 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.299158 4926 scope.go:122] "RemoveContainer" containerID="05cfd4fb21377a75dd0b48285493154c297d94a6421f5f7f84c4dd3411b75e14" Oct 30 21:26:52.317502 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.317469 4926 scope.go:122] "RemoveContainer" containerID="05cfd4fb21377a75dd0b48285493154c297d94a6421f5f7f84c4dd3411b75e14" Oct 30 21:26:52.317872 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:26:52.317838 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05cfd4fb21377a75dd0b48285493154c297d94a6421f5f7f84c4dd3411b75e14\": container with ID starting with 05cfd4fb21377a75dd0b48285493154c297d94a6421f5f7f84c4dd3411b75e14 not found: ID does not exist" containerID="05cfd4fb21377a75dd0b48285493154c297d94a6421f5f7f84c4dd3411b75e14" Oct 30 21:26:52.317986 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.317882 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05cfd4fb21377a75dd0b48285493154c297d94a6421f5f7f84c4dd3411b75e14"} err="failed to get container status \"05cfd4fb21377a75dd0b48285493154c297d94a6421f5f7f84c4dd3411b75e14\": rpc error: code = NotFound desc = could not find container \"05cfd4fb21377a75dd0b48285493154c297d94a6421f5f7f84c4dd3411b75e14\": container with ID starting with 05cfd4fb21377a75dd0b48285493154c297d94a6421f5f7f84c4dd3411b75e14 not found: ID does not exist" Oct 30 21:26:52.354546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.354514 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:52.354546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.354543 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtvfd\" (UniqueName: \"kubernetes.io/projected/0e6eca6b-e95a-44c5-bc67-f431a266729d-kube-api-access-rtvfd\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:52.354546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.354553 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:52.354764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.354562 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:52.354764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.354571 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:52.354764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.354578 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6eca6b-e95a-44c5-bc67-f431a266729d-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:52.354764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.354587 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6eca6b-e95a-44c5-bc67-f431a266729d-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:52.360095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.360063 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z"] Oct 30 21:26:52.364527 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.364501 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-4f27z"] Oct 30 21:26:52.595527 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.595492 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx"] Oct 30 21:26:52.597017 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.596987 4926 cpu_manager.go:414] "containerMap: removing container" podUID="b30f1ce4-1915-46fa-8393-9e3ffbf922ea" containerName="revoke-keys" Oct 30 21:26:52.597187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.597172 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b30f1ce4-1915-46fa-8393-9e3ffbf922ea" containerName="revoke-keys" Oct 30 21:26:52.597318 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.597307 4926 cpu_manager.go:414] "containerMap: removing container" podUID="226ee133-aef2-4cd2-b0bb-5bb745a03862" containerName="istio-proxy" Oct 30 21:26:52.597407 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.597397 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="226ee133-aef2-4cd2-b0bb-5bb745a03862" containerName="istio-proxy" Oct 30 21:26:52.597523 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.597513 4926 cpu_manager.go:414] "containerMap: removing container" podUID="0e6eca6b-e95a-44c5-bc67-f431a266729d" containerName="istio-proxy" Oct 30 21:26:52.597593 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.597584 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e6eca6b-e95a-44c5-bc67-f431a266729d" containerName="istio-proxy" Oct 30 21:26:52.597826 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.597813 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="226ee133-aef2-4cd2-b0bb-5bb745a03862" containerName="istio-proxy" Oct 30 21:26:52.597932 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.597921 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e6eca6b-e95a-44c5-bc67-f431a266729d" containerName="istio-proxy" Oct 30 21:26:52.598020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.598010 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="b30f1ce4-1915-46fa-8393-9e3ffbf922ea" containerName="revoke-keys" Oct 30 21:26:52.648054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.646192 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx"] Oct 30 21:26:52.648054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.647460 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.663014 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.662972 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk"] Oct 30 21:26:52.671682 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.671653 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-ca18b3-openshift-default-68c46d7b5-lvlsk"] Oct 30 21:26:52.766653 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.766560 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-workload-certs\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.766843 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.766715 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae78b55-4403-42fc-8646-633c7bbded37-istio-podinfo\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.766843 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.766789 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-istio-envoy\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.766976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.766870 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-credential-socket\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.766976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.766914 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-istio-data\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.767109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.767044 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae78b55-4403-42fc-8646-633c7bbded37-istio-token\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.767109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.767078 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae78b55-4403-42fc-8646-633c7bbded37-istiod-ca-cert\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.767223 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.767108 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-workload-socket\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.767279 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.767245 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h4tw\" (UniqueName: \"kubernetes.io/projected/eae78b55-4403-42fc-8646-633c7bbded37-kube-api-access-4h4tw\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.869097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.869050 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-workload-certs\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.869346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.869176 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae78b55-4403-42fc-8646-633c7bbded37-istio-podinfo\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.869346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.869219 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-istio-envoy\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.869346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.869270 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-credential-socket\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.869346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.869298 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-istio-data\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.869600 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.869378 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae78b55-4403-42fc-8646-633c7bbded37-istio-token\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.869600 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.869411 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae78b55-4403-42fc-8646-633c7bbded37-istiod-ca-cert\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.869600 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.869444 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-workload-socket\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.869600 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.869541 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h4tw\" (UniqueName: \"kubernetes.io/projected/eae78b55-4403-42fc-8646-633c7bbded37-kube-api-access-4h4tw\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.870468 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.870443 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-credential-socket\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.871153 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.871107 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-workload-certs\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.871428 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.871392 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-workload-socket\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.871525 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.871501 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-istio-data\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.871653 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.871618 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae78b55-4403-42fc-8646-633c7bbded37-istiod-ca-cert\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.873585 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.873565 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-istio-envoy\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.874163 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.874144 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae78b55-4403-42fc-8646-633c7bbded37-istio-podinfo\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.892711 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.892665 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae78b55-4403-42fc-8646-633c7bbded37-istio-token\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.892921 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.892900 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h4tw\" (UniqueName: \"kubernetes.io/projected/eae78b55-4403-42fc-8646-633c7bbded37-kube-api-access-4h4tw\") pod \"e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:52.976647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:52.976611 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:53.022627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.022541 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e6eca6b-e95a-44c5-bc67-f431a266729d" path="/var/lib/kubelet/pods/0e6eca6b-e95a-44c5-bc67-f431a266729d/volumes" Oct 30 21:26:53.023359 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.023335 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="226ee133-aef2-4cd2-b0bb-5bb745a03862" path="/var/lib/kubelet/pods/226ee133-aef2-4cd2-b0bb-5bb745a03862/volumes" Oct 30 21:26:53.307048 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.307008 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc"] Oct 30 21:26:53.312568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.312541 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.315642 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.315148 4926 generic.go:356] "Generic (PLEG): container finished" podID="76bddd4e-d53f-41f6-95a5-bc445d3eedcf" containerID="a4e7592dd1ff4fbb169797f585775c45b5a7b6e2b55bd3a64890ee48f2d691a0" exitCode=0 Oct 30 21:26:53.315642 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.315228 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" event={"ID":"76bddd4e-d53f-41f6-95a5-bc445d3eedcf","Type":"ContainerDied","Data":"a4e7592dd1ff4fbb169797f585775c45b5a7b6e2b55bd3a64890ee48f2d691a0"} Oct 30 21:26:53.316107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.315789 4926 scope.go:122] "RemoveContainer" containerID="a4e7592dd1ff4fbb169797f585775c45b5a7b6e2b55bd3a64890ee48f2d691a0" Oct 30 21:26:53.318572 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.318539 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:26:53.318750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.318728 4926 generic.go:356] "Generic (PLEG): container finished" podID="4a6c48f2-e476-4ae9-82b4-b476a225350d" containerID="8c9373da8610358a4fd85e0238b3633925c6510a2798c31c497d3bee985b870d" exitCode=0 Oct 30 21:26:53.318860 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.318743 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:26:53.318860 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.318793 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:26:53.318860 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.318803 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" event={"ID":"4a6c48f2-e476-4ae9-82b4-b476a225350d","Type":"ContainerDied","Data":"8c9373da8610358a4fd85e0238b3633925c6510a2798c31c497d3bee985b870d"} Oct 30 21:26:53.336133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.336067 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx"] Oct 30 21:26:53.352587 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.352559 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-79c24a-openshift-default-dockercfg-k6ggn\"" Oct 30 21:26:53.355851 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.355816 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc"] Oct 30 21:26:53.368677 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.367765 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:26:53.368677 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.367855 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:26:53.368677 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.367892 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:26:53.379008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.378966 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-podinfo\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.379446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.379419 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnv4v\" (UniqueName: \"kubernetes.io/projected/84b972f9-f7bd-4895-9863-dbf951a09e00-kube-api-access-vnv4v\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.379544 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.379505 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b972f9-f7bd-4895-9863-dbf951a09e00-istiod-ca-cert\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.379608 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.379545 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-workload-certs\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.379608 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.379582 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-data\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.379717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.379611 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-credential-socket\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.379854 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.379822 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-workload-socket\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.379939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.379880 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-token\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.379939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.379919 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-envoy\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.484064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.482116 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-podinfo\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.484064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.482208 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnv4v\" (UniqueName: \"kubernetes.io/projected/84b972f9-f7bd-4895-9863-dbf951a09e00-kube-api-access-vnv4v\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.484064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.482255 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b972f9-f7bd-4895-9863-dbf951a09e00-istiod-ca-cert\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.484064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.482290 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-workload-certs\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.484064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.482327 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-data\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.484064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.482357 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-credential-socket\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.484064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.482487 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-workload-socket\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.484064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.482528 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-token\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.484064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.482564 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-envoy\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.484064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.483437 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-workload-certs\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.484735 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.484477 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b972f9-f7bd-4895-9863-dbf951a09e00-istiod-ca-cert\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.485147 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.485012 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-credential-socket\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.486267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.485394 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-data\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.486267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.485496 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-workload-socket\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.504997 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.504932 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-podinfo\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.509140 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.509083 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-envoy\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.515090 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.515054 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnv4v\" (UniqueName: \"kubernetes.io/projected/84b972f9-f7bd-4895-9863-dbf951a09e00-kube-api-access-vnv4v\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.554928 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.554864 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-token\") pod \"e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:53.645437 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:53.632791 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:54.091824 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.091792 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:26:54.101762 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.101667 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.251828 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-envoy\") pod \"4a6c48f2-e476-4ae9-82b4-b476a225350d\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.251916 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-workload-certs\") pod \"4a6c48f2-e476-4ae9-82b4-b476a225350d\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.251993 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-data\") pod \"4a6c48f2-e476-4ae9-82b4-b476a225350d\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.252021 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a6c48f2-e476-4ae9-82b4-b476a225350d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a6c48f2-e476-4ae9-82b4-b476a225350d-istiod-ca-cert\") pod \"4a6c48f2-e476-4ae9-82b4-b476a225350d\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.252074 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-credential-socket\") pod \"472a982c-6acb-4164-96f6-5cf2904a4c8d\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.252164 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/472a982c-6acb-4164-96f6-5cf2904a4c8d-kube-api-access-mhcwv\" (UniqueName: \"kubernetes.io/projected/472a982c-6acb-4164-96f6-5cf2904a4c8d-kube-api-access-mhcwv\") pod \"472a982c-6acb-4164-96f6-5cf2904a4c8d\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.252270 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-token\" (UniqueName: \"kubernetes.io/projected/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-token\") pod \"4a6c48f2-e476-4ae9-82b4-b476a225350d\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.252329 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-envoy\") pod \"472a982c-6acb-4164-96f6-5cf2904a4c8d\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.252385 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-token\" (UniqueName: \"kubernetes.io/projected/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-token\") pod \"472a982c-6acb-4164-96f6-5cf2904a4c8d\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.252416 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-workload-socket\") pod \"4a6c48f2-e476-4ae9-82b4-b476a225350d\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.252455 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a6c48f2-e476-4ae9-82b4-b476a225350d-kube-api-access-856mh\" (UniqueName: \"kubernetes.io/projected/4a6c48f2-e476-4ae9-82b4-b476a225350d-kube-api-access-856mh\") pod \"4a6c48f2-e476-4ae9-82b4-b476a225350d\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.252483 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/472a982c-6acb-4164-96f6-5cf2904a4c8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/472a982c-6acb-4164-96f6-5cf2904a4c8d-istiod-ca-cert\") pod \"472a982c-6acb-4164-96f6-5cf2904a4c8d\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.252574 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-podinfo\") pod \"4a6c48f2-e476-4ae9-82b4-b476a225350d\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.252645 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-credential-socket\") pod \"4a6c48f2-e476-4ae9-82b4-b476a225350d\" (UID: \"4a6c48f2-e476-4ae9-82b4-b476a225350d\") " Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.252716 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-workload-socket\") pod \"472a982c-6acb-4164-96f6-5cf2904a4c8d\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.252758 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-podinfo\") pod \"472a982c-6acb-4164-96f6-5cf2904a4c8d\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.252787 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-workload-certs\") pod \"472a982c-6acb-4164-96f6-5cf2904a4c8d\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.252818 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-data\") pod \"472a982c-6acb-4164-96f6-5cf2904a4c8d\" (UID: \"472a982c-6acb-4164-96f6-5cf2904a4c8d\") " Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.259638 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-data" pod "4a6c48f2-e476-4ae9-82b4-b476a225350d" (UID: "4a6c48f2-e476-4ae9-82b4-b476a225350d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.262653 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a6c48f2-e476-4ae9-82b4-b476a225350d-istiod-ca-cert" pod "4a6c48f2-e476-4ae9-82b4-b476a225350d" (UID: "4a6c48f2-e476-4ae9-82b4-b476a225350d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.263004 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-credential-socket" pod "472a982c-6acb-4164-96f6-5cf2904a4c8d" (UID: "472a982c-6acb-4164-96f6-5cf2904a4c8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.272077 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/472a982c-6acb-4164-96f6-5cf2904a4c8d-istiod-ca-cert" pod "472a982c-6acb-4164-96f6-5cf2904a4c8d" (UID: "472a982c-6acb-4164-96f6-5cf2904a4c8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.276485 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-token" pod "4a6c48f2-e476-4ae9-82b4-b476a225350d" (UID: "4a6c48f2-e476-4ae9-82b4-b476a225350d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.279318 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-envoy" pod "472a982c-6acb-4164-96f6-5cf2904a4c8d" (UID: "472a982c-6acb-4164-96f6-5cf2904a4c8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.282913 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-data" pod "472a982c-6acb-4164-96f6-5cf2904a4c8d" (UID: "472a982c-6acb-4164-96f6-5cf2904a4c8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.287305 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-workload-socket" pod "4a6c48f2-e476-4ae9-82b4-b476a225350d" (UID: "4a6c48f2-e476-4ae9-82b4-b476a225350d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.290336 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a6c48f2-e476-4ae9-82b4-b476a225350d-kube-api-access-856mh" pod "4a6c48f2-e476-4ae9-82b4-b476a225350d" (UID: "4a6c48f2-e476-4ae9-82b4-b476a225350d"). InnerVolumeSpecName "kube-api-access-856mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.291775 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-podinfo" pod "4a6c48f2-e476-4ae9-82b4-b476a225350d" (UID: "4a6c48f2-e476-4ae9-82b4-b476a225350d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.292013 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-workload-certs" pod "472a982c-6acb-4164-96f6-5cf2904a4c8d" (UID: "472a982c-6acb-4164-96f6-5cf2904a4c8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.292181 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472a982c-6acb-4164-96f6-5cf2904a4c8d-kube-api-access-mhcwv" pod "472a982c-6acb-4164-96f6-5cf2904a4c8d" (UID: "472a982c-6acb-4164-96f6-5cf2904a4c8d"). InnerVolumeSpecName "kube-api-access-mhcwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.292375 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-credential-socket" pod "4a6c48f2-e476-4ae9-82b4-b476a225350d" (UID: "4a6c48f2-e476-4ae9-82b4-b476a225350d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.292535 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-workload-socket" pod "472a982c-6acb-4164-96f6-5cf2904a4c8d" (UID: "472a982c-6acb-4164-96f6-5cf2904a4c8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.293123 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-podinfo" pod "472a982c-6acb-4164-96f6-5cf2904a4c8d" (UID: "472a982c-6acb-4164-96f6-5cf2904a4c8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:26:54.298634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.294204 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-envoy" pod "4a6c48f2-e476-4ae9-82b4-b476a225350d" (UID: "4a6c48f2-e476-4ae9-82b4-b476a225350d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:54.300386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.298959 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-token" pod "472a982c-6acb-4164-96f6-5cf2904a4c8d" (UID: "472a982c-6acb-4164-96f6-5cf2904a4c8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:26:54.300646 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.300607 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-workload-certs" pod "4a6c48f2-e476-4ae9-82b4-b476a225350d" (UID: "4a6c48f2-e476-4ae9-82b4-b476a225350d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:26:54.334076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.333993 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" event={"ID":"eae78b55-4403-42fc-8646-633c7bbded37","Type":"ContainerStarted","Data":"fb4e93836eede910d5e9e4b8d70185659de098f4a23a464bc5ca76499659c315"} Oct 30 21:26:54.334269 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.334093 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" event={"ID":"eae78b55-4403-42fc-8646-633c7bbded37","Type":"ContainerStarted","Data":"04fafa254197f6d25876d2c5227689a6b6320b6dee9a513b03113e70a123105d"} Oct 30 21:26:54.336764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.336702 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" event={"ID":"76bddd4e-d53f-41f6-95a5-bc445d3eedcf","Type":"ContainerStarted","Data":"9611169f6e7e5447bd81d6281071d4bd8894bec654e1f060ec40271cfdc9e4ae"} Oct 30 21:26:54.338388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.338360 4926 generic.go:356] "Generic (PLEG): container finished" podID="472a982c-6acb-4164-96f6-5cf2904a4c8d" containerID="5a9de4e8e86b80d8baaebb07cf93d204ba0ce046332869c8352c24fa936029fb" exitCode=0 Oct 30 21:26:54.338503 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.338406 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" Oct 30 21:26:54.338503 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.338445 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" event={"ID":"472a982c-6acb-4164-96f6-5cf2904a4c8d","Type":"ContainerDied","Data":"5a9de4e8e86b80d8baaebb07cf93d204ba0ce046332869c8352c24fa936029fb"} Oct 30 21:26:54.338614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.338501 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf" event={"ID":"472a982c-6acb-4164-96f6-5cf2904a4c8d","Type":"ContainerDied","Data":"598f267559a43ee780f8a61da595b61f91383af60b9b987af49788a2a5a92626"} Oct 30 21:26:54.338614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.338548 4926 scope.go:122] "RemoveContainer" containerID="5a9de4e8e86b80d8baaebb07cf93d204ba0ce046332869c8352c24fa936029fb" Oct 30 21:26:54.340765 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.340633 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" Oct 30 21:26:54.340765 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.340681 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" event={"ID":"4a6c48f2-e476-4ae9-82b4-b476a225350d","Type":"ContainerDied","Data":"cfb0c043ab223cbeb15845e4c7ceb6679c680f3e9a1dba0a8e6dabc49371cba9"} Oct 30 21:26:54.356796 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.356775 4926 scope.go:122] "RemoveContainer" containerID="5a9de4e8e86b80d8baaebb07cf93d204ba0ce046332869c8352c24fa936029fb" Oct 30 21:26:54.373556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.367504 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:54.373556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.367526 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:54.373556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.367534 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:54.373556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.367543 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a6c48f2-e476-4ae9-82b4-b476a225350d-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:54.373556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.367553 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:54.373556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.367566 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhcwv\" (UniqueName: \"kubernetes.io/projected/472a982c-6acb-4164-96f6-5cf2904a4c8d-kube-api-access-mhcwv\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:54.373556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.367579 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:54.373556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.367592 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:54.373556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.367603 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:54.373556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.367615 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:54.373556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.367626 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-856mh\" (UniqueName: \"kubernetes.io/projected/4a6c48f2-e476-4ae9-82b4-b476a225350d-kube-api-access-856mh\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:54.373556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.367639 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/472a982c-6acb-4164-96f6-5cf2904a4c8d-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:54.373556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.367653 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a6c48f2-e476-4ae9-82b4-b476a225350d-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:54.373556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.367666 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a6c48f2-e476-4ae9-82b4-b476a225350d-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:54.373556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.367674 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:54.373556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.367685 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:54.373556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.367692 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:54.373556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.367700 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/472a982c-6acb-4164-96f6-5cf2904a4c8d-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:26:54.373556 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:26:54.369784 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a9de4e8e86b80d8baaebb07cf93d204ba0ce046332869c8352c24fa936029fb\": container with ID starting with 5a9de4e8e86b80d8baaebb07cf93d204ba0ce046332869c8352c24fa936029fb not found: ID does not exist" containerID="5a9de4e8e86b80d8baaebb07cf93d204ba0ce046332869c8352c24fa936029fb" Oct 30 21:26:54.373556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.369818 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a9de4e8e86b80d8baaebb07cf93d204ba0ce046332869c8352c24fa936029fb"} err="failed to get container status \"5a9de4e8e86b80d8baaebb07cf93d204ba0ce046332869c8352c24fa936029fb\": rpc error: code = NotFound desc = could not find container \"5a9de4e8e86b80d8baaebb07cf93d204ba0ce046332869c8352c24fa936029fb\": container with ID starting with 5a9de4e8e86b80d8baaebb07cf93d204ba0ce046332869c8352c24fa936029fb not found: ID does not exist" Oct 30 21:26:54.373556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.369839 4926 scope.go:122] "RemoveContainer" containerID="8c9373da8610358a4fd85e0238b3633925c6510a2798c31c497d3bee985b870d" Oct 30 21:26:54.382988 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.382932 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" podStartSLOduration=2.382916998 podStartE2EDuration="2.382916998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:54.379304947 +0000 UTC m=+1601.500378693" watchObservedRunningTime="2026-10-30 21:26:54.382916998 +0000 UTC m=+1601.503990744" Oct 30 21:26:54.494845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.481360 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:26:54.494845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.481420 4926 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:26:54.494845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.481489 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" podUID="76bddd4e-d53f-41f6-95a5-bc445d3eedcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 21:26:54.494845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.492937 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:26:54.494845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.493111 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:26:54.494845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.493250 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:26:54.506198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.506166 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc"] Oct 30 21:26:54.516766 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.516725 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf"] Oct 30 21:26:54.526490 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.526461 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-j6lqf"] Oct 30 21:26:54.548375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.547071 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj"] Oct 30 21:26:54.548375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.547133 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj"] Oct 30 21:26:54.976984 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.976952 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:26:54.978723 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.978682 4926 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 21:26:54.978869 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:54.978766 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" podUID="eae78b55-4403-42fc-8646-633c7bbded37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 21:26:55.018873 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.018829 4926 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:26:55.019045 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.018897 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-8tphj" podUID="4a6c48f2-e476-4ae9-82b4-b476a225350d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:55.021545 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.021520 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="472a982c-6acb-4164-96f6-5cf2904a4c8d" path="/var/lib/kubelet/pods/472a982c-6acb-4164-96f6-5cf2904a4c8d/volumes" Oct 30 21:26:55.022120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.022104 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a6c48f2-e476-4ae9-82b4-b476a225350d" path="/var/lib/kubelet/pods/4a6c48f2-e476-4ae9-82b4-b476a225350d/volumes" Oct 30 21:26:55.037010 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.036985 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:26:55.357161 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.357027 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" event={"ID":"84b972f9-f7bd-4895-9863-dbf951a09e00","Type":"ContainerStarted","Data":"1f6c8e587e481b029dd7af5a4d048f2d700920324b7ec3fea6d5be1a7688f540"} Oct 30 21:26:55.357161 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.357133 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" event={"ID":"84b972f9-f7bd-4895-9863-dbf951a09e00","Type":"ContainerStarted","Data":"96971bb47d9ab3f95d7b4720a19019c649db4f15b0cd821f4a4bdba0abc6bbf6"} Oct 30 21:26:55.428102 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.422746 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" podStartSLOduration=2.422697106 podStartE2EDuration="2.422697106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:26:55.409976377 +0000 UTC m=+1602.531050124" watchObservedRunningTime="2026-10-30 21:26:55.422697106 +0000 UTC m=+1602.543770845" Oct 30 21:26:55.509057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.475170 4926 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:26:55.509057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.475233 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" podUID="76bddd4e-d53f-41f6-95a5-bc445d3eedcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 21:26:55.644861 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.644752 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:26:55.646398 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.646345 4926 patch_prober.go:29] interesting pod/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:26:55.646547 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.646436 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" podUID="84b972f9-f7bd-4895-9863-dbf951a09e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 21:26:55.941812 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.941771 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6"] Oct 30 21:26:55.954061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.943089 4926 cpu_manager.go:414] "containerMap: removing container" podUID="472a982c-6acb-4164-96f6-5cf2904a4c8d" containerName="istio-proxy" Oct 30 21:26:55.954329 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.954294 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="472a982c-6acb-4164-96f6-5cf2904a4c8d" containerName="istio-proxy" Oct 30 21:26:55.954530 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.954518 4926 cpu_manager.go:414] "containerMap: removing container" podUID="4a6c48f2-e476-4ae9-82b4-b476a225350d" containerName="istio-proxy" Oct 30 21:26:55.954624 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.954611 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a6c48f2-e476-4ae9-82b4-b476a225350d" containerName="istio-proxy" Oct 30 21:26:55.954958 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.954942 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="472a982c-6acb-4164-96f6-5cf2904a4c8d" containerName="istio-proxy" Oct 30 21:26:55.955108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.955094 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a6c48f2-e476-4ae9-82b4-b476a225350d" containerName="istio-proxy" Oct 30 21:26:55.979507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.979455 4926 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 21:26:55.980135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.980084 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" podUID="eae78b55-4403-42fc-8646-633c7bbded37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 21:26:55.980888 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:55.980862 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" Oct 30 21:26:56.027058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:56.001531 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-79e911-serving-cert\"" Oct 30 21:26:56.127204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:56.126629 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e44f1d5-29f6-4129-b244-82dd24a733b8-metrics-certs\") pod \"maas-api-e2e-inf-a-79e911-54988765fb-lkkb6\" (UID: \"3e44f1d5-29f6-4129-b244-82dd24a733b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" Oct 30 21:26:56.127204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:56.126738 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e44f1d5-29f6-4129-b244-82dd24a733b8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-79e911-54988765fb-lkkb6\" (UID: \"3e44f1d5-29f6-4129-b244-82dd24a733b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" Oct 30 21:26:56.127204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:56.126927 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgnv4\" (UniqueName: \"kubernetes.io/projected/3e44f1d5-29f6-4129-b244-82dd24a733b8-kube-api-access-jgnv4\") pod \"maas-api-e2e-inf-a-79e911-54988765fb-lkkb6\" (UID: \"3e44f1d5-29f6-4129-b244-82dd24a733b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" Oct 30 21:26:56.127204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:56.127051 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e44f1d5-29f6-4129-b244-82dd24a733b8-maas-api-tls\") pod \"maas-api-e2e-inf-a-79e911-54988765fb-lkkb6\" (UID: \"3e44f1d5-29f6-4129-b244-82dd24a733b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" Oct 30 21:26:56.254072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:56.237462 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e44f1d5-29f6-4129-b244-82dd24a733b8-metrics-certs\") pod \"maas-api-e2e-inf-a-79e911-54988765fb-lkkb6\" (UID: \"3e44f1d5-29f6-4129-b244-82dd24a733b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" Oct 30 21:26:56.254072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:56.237568 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e44f1d5-29f6-4129-b244-82dd24a733b8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-79e911-54988765fb-lkkb6\" (UID: \"3e44f1d5-29f6-4129-b244-82dd24a733b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" Oct 30 21:26:56.254072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:56.237732 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgnv4\" (UniqueName: \"kubernetes.io/projected/3e44f1d5-29f6-4129-b244-82dd24a733b8-kube-api-access-jgnv4\") pod \"maas-api-e2e-inf-a-79e911-54988765fb-lkkb6\" (UID: \"3e44f1d5-29f6-4129-b244-82dd24a733b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" Oct 30 21:26:56.254072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:56.237833 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e44f1d5-29f6-4129-b244-82dd24a733b8-maas-api-tls\") pod \"maas-api-e2e-inf-a-79e911-54988765fb-lkkb6\" (UID: \"3e44f1d5-29f6-4129-b244-82dd24a733b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" Oct 30 21:26:56.283058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:56.257586 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6"] Oct 30 21:26:56.381054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:56.317820 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgnv4\" (UniqueName: \"kubernetes.io/projected/3e44f1d5-29f6-4129-b244-82dd24a733b8-kube-api-access-jgnv4\") pod \"maas-api-e2e-inf-a-79e911-54988765fb-lkkb6\" (UID: \"3e44f1d5-29f6-4129-b244-82dd24a733b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" Oct 30 21:26:56.381054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:56.346524 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e44f1d5-29f6-4129-b244-82dd24a733b8-openshift-service-ca\") pod \"maas-api-e2e-inf-a-79e911-54988765fb-lkkb6\" (UID: \"3e44f1d5-29f6-4129-b244-82dd24a733b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" Oct 30 21:26:56.491649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:56.491543 4926 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:26:56.560090 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:56.491888 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" podUID="76bddd4e-d53f-41f6-95a5-bc445d3eedcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 21:26:56.560090 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:56.543764 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e44f1d5-29f6-4129-b244-82dd24a733b8-metrics-certs\") pod \"maas-api-e2e-inf-a-79e911-54988765fb-lkkb6\" (UID: \"3e44f1d5-29f6-4129-b244-82dd24a733b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" Oct 30 21:26:56.639249 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:56.637643 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e44f1d5-29f6-4129-b244-82dd24a733b8-maas-api-tls\") pod \"maas-api-e2e-inf-a-79e911-54988765fb-lkkb6\" (UID: \"3e44f1d5-29f6-4129-b244-82dd24a733b8\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" Oct 30 21:26:56.647574 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:56.647529 4926 patch_prober.go:29] interesting pod/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:26:56.647870 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:56.647827 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" podUID="84b972f9-f7bd-4895-9863-dbf951a09e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 21:26:56.720854 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:56.701890 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" Oct 30 21:26:56.924194 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:56.919148 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:56.980067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:56.978421 4926 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 21:26:56.980067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:56.978528 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" podUID="eae78b55-4403-42fc-8646-633c7bbded37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 21:26:57.249058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:57.248146 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:57.460110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:57.427205 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:57.478268 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:57.467451 4926 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:26:57.478268 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:57.467517 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" podUID="76bddd4e-d53f-41f6-95a5-bc445d3eedcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 21:26:57.841635 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:57.661387 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:57.841635 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:57.662678 4926 patch_prober.go:29] interesting pod/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:26:57.841635 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:57.662724 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" podUID="84b972f9-f7bd-4895-9863-dbf951a09e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 21:26:57.841635 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:57.773348 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:57.841635 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:57.795163 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:57.841635 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:57.795623 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 21:26:57.841635 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:57.796934 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:57.841635 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:57.840150 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:57.921053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:57.908714 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:26:57.921053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:57.920088 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:57.921053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:57.920638 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:26:57.964489 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:57.958753 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:58.002251 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:57.988616 4926 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 21:26:58.002251 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:57.988674 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" podUID="eae78b55-4403-42fc-8646-633c7bbded37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 21:26:58.142150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:58.056115 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:58.302673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:58.290945 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 21:26:58.483090 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:58.474152 4926 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:26:58.483090 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:58.474216 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" podUID="76bddd4e-d53f-41f6-95a5-bc445d3eedcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 21:26:58.556080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:58.556004 4926 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:26:58.595888 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:58.595850 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6"] Oct 30 21:26:58.606403 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:58.605418 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:58.689122 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:58.663144 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 21:26:58.726964 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:58.726930 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:26:58.728503 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:58.728475 4926 patch_prober.go:29] interesting pod/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:26:58.728717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:58.728680 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" podUID="84b972f9-f7bd-4895-9863-dbf951a09e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 21:26:58.893314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:58.865228 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:58.893314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:58.865474 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:58.893314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:58.865639 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 21:26:58.893314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:58.865884 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:26:58.893314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:58.866128 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:26:58.893314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:58.866183 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:26:58.893314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:58.866222 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:26:58.893314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:58.866259 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:26:59.051104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.045151 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:59.094023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.074632 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 21:26:59.094023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.076472 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:26:59.144056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.119354 4926 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 21:26:59.144056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.119419 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" podUID="eae78b55-4403-42fc-8646-633c7bbded37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 21:26:59.265592 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.249778 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:26:59.300477 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.300433 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/health\": context deadline exceeded" Oct 30 21:26:59.387273 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.387170 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:59.595626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.595581 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" event={"ID":"3e44f1d5-29f6-4129-b244-82dd24a733b8","Type":"ContainerStarted","Data":"8debe2ac5c507ac75805b7e93de75e27506e4622e582e9b45f73de8943558c71"} Oct 30 21:26:59.595989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.595917 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:26:59.597139 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.597122 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:26:59.597321 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.597306 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:26:59.641360 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.641252 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:59.642156 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.642128 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:26:59.658058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.649556 4926 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:26:59.658058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.649613 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" podUID="76bddd4e-d53f-41f6-95a5-bc445d3eedcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 21:26:59.658058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.650492 4926 patch_prober.go:29] interesting pod/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:26:59.658058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.650535 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" podUID="84b972f9-f7bd-4895-9863-dbf951a09e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 21:26:59.692344 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.692300 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:59.887384 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.871772 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:26:59.887384 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.885712 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 21:26:59.887384 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.886101 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 21:26:59.887384 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:26:59.886385 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 21:27:00.080838 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:00.080685 4926 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 21:27:00.081453 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:00.081418 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" podUID="eae78b55-4403-42fc-8646-633c7bbded37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 21:27:00.088601 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:00.082961 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:00.160055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:00.154843 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:27:00.293275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:00.293219 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:00.324094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:00.293627 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:00.324094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:00.304526 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:27:00.459057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:00.458367 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": context deadline exceeded" Oct 30 21:27:00.482210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:00.482155 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:00.508285 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:00.501767 4926 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:27:00.508285 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:00.501827 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" podUID="76bddd4e-d53f-41f6-95a5-bc445d3eedcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 21:27:00.543059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:00.537254 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:00.655860 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:00.655821 4926 patch_prober.go:29] interesting pod/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:27:00.656168 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:00.656140 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" podUID="84b972f9-f7bd-4895-9863-dbf951a09e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 21:27:00.709139 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:00.709077 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:00.891056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:00.885348 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:00.891056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:00.886273 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:00.891056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:00.887120 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:00.891056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:00.887509 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 21:27:00.891056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:00.887790 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 21:27:01.088446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.088326 4926 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 21:27:01.136218 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.136169 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" podUID="eae78b55-4403-42fc-8646-633c7bbded37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 21:27:01.136988 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.090356 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:01.311059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.308688 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:01.311059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.309301 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 30 21:27:01.494141 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.366597 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:01.494141 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.458970 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:27:01.494141 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.459193 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:01.494141 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.468643 4926 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:27:01.494141 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.468703 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" podUID="76bddd4e-d53f-41f6-95a5-bc445d3eedcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 21:27:01.529058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.504127 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:01.529058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.504576 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:27:01.556242 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.541534 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:01.556242 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.542165 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:27:01.683391 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.683354 4926 patch_prober.go:29] interesting pod/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:27:01.683653 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.683629 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" podUID="84b972f9-f7bd-4895-9863-dbf951a09e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 21:27:01.712058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.709632 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:01.831328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.831204 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": context deadline exceeded" Oct 30 21:27:01.898073 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.893176 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:01.898073 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.893577 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": context deadline exceeded" Oct 30 21:27:01.898073 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.893830 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded" Oct 30 21:27:01.906478 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.906437 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 21:27:01.907090 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.907016 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 21:27:01.908073 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.908053 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:27:01.990377 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.990339 4926 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 21:27:01.990649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:01.990623 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" podUID="eae78b55-4403-42fc-8646-633c7bbded37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 21:27:02.079544 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.076374 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:27:02.090325 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.090184 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:02.199211 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.170972 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:02.268630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.255793 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" event={"ID":"3e44f1d5-29f6-4129-b244-82dd24a733b8","Type":"ContainerStarted","Data":"9a74dd74eaa7130369f442ebcc73fcc5285963eb509e047ea1114e3dfef1681c"} Oct 30 21:27:02.268630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.255859 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" Oct 30 21:27:02.292648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.292602 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:02.309934 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.309893 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:02.499890 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.499857 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:27:02.501144 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.310227 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:02.502899 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.310257 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 30 21:27:02.511590 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.469404 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:02.537787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.472023 4926 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:27:02.537787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.512846 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" podUID="76bddd4e-d53f-41f6-95a5-bc445d3eedcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 21:27:02.537787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.383606 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:27:02.537787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.512013 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:27:02.537787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.512088 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:02.537787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.537119 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:02.553166 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.553118 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 21:27:02.577474 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.577436 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr"] Oct 30 21:27:02.786111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.634611 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:02.786111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.674166 4926 patch_prober.go:29] interesting pod/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 21:27:02.786111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.780348 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" podUID="84b972f9-f7bd-4895-9863-dbf951a09e00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 21:27:02.786111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.722152 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Oct 30 21:27:02.902060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.899486 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:02.914067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.906887 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 21:27:02.914067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.907173 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 21:27:02.914067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.907831 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:02.970622 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.970584 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr"] Oct 30 21:27:02.970965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.970945 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:27:02.971150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.971136 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:27:02.972176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.971928 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" Oct 30 21:27:02.994363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.992489 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c5145c-serving-cert\"" Oct 30 21:27:02.994363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.992601 4926 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Oct 30 21:27:02.994363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:02.992647 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" podUID="eae78b55-4403-42fc-8646-633c7bbded37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Oct 30 21:27:03.040361 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.038323 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c79ad431-18a6-4301-bb75-db4a80b01755-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr\" (UID: \"c79ad431-18a6-4301-bb75-db4a80b01755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" Oct 30 21:27:03.040361 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.038489 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffrrc\" (UniqueName: \"kubernetes.io/projected/c79ad431-18a6-4301-bb75-db4a80b01755-kube-api-access-ffrrc\") pod \"maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr\" (UID: \"c79ad431-18a6-4301-bb75-db4a80b01755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" Oct 30 21:27:03.040361 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.038702 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c79ad431-18a6-4301-bb75-db4a80b01755-metrics-certs\") pod \"maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr\" (UID: \"c79ad431-18a6-4301-bb75-db4a80b01755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" Oct 30 21:27:03.040361 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.038850 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c79ad431-18a6-4301-bb75-db4a80b01755-maas-api-tls\") pod \"maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr\" (UID: \"c79ad431-18a6-4301-bb75-db4a80b01755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" Oct 30 21:27:03.125729 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.118689 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:03.149189 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.149147 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c79ad431-18a6-4301-bb75-db4a80b01755-metrics-certs\") pod \"maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr\" (UID: \"c79ad431-18a6-4301-bb75-db4a80b01755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" Oct 30 21:27:03.149622 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.149578 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c79ad431-18a6-4301-bb75-db4a80b01755-maas-api-tls\") pod \"maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr\" (UID: \"c79ad431-18a6-4301-bb75-db4a80b01755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" Oct 30 21:27:03.149942 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.149921 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c79ad431-18a6-4301-bb75-db4a80b01755-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr\" (UID: \"c79ad431-18a6-4301-bb75-db4a80b01755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" Oct 30 21:27:03.150262 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.150231 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffrrc\" (UniqueName: \"kubernetes.io/projected/c79ad431-18a6-4301-bb75-db4a80b01755-kube-api-access-ffrrc\") pod \"maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr\" (UID: \"c79ad431-18a6-4301-bb75-db4a80b01755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" Oct 30 21:27:03.250088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.222680 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c79ad431-18a6-4301-bb75-db4a80b01755-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr\" (UID: \"c79ad431-18a6-4301-bb75-db4a80b01755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" Oct 30 21:27:03.250088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.229662 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c79ad431-18a6-4301-bb75-db4a80b01755-metrics-certs\") pod \"maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr\" (UID: \"c79ad431-18a6-4301-bb75-db4a80b01755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" Oct 30 21:27:03.250088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.246909 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffrrc\" (UniqueName: \"kubernetes.io/projected/c79ad431-18a6-4301-bb75-db4a80b01755-kube-api-access-ffrrc\") pod \"maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr\" (UID: \"c79ad431-18a6-4301-bb75-db4a80b01755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" Oct 30 21:27:03.320470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.320384 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c79ad431-18a6-4301-bb75-db4a80b01755-maas-api-tls\") pod \"maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr\" (UID: \"c79ad431-18a6-4301-bb75-db4a80b01755\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" Oct 30 21:27:03.465121 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.450569 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:27:03.465451 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.465427 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:27:03.465579 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.465568 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:27:03.521055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.511803 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:03.521055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.514375 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 30 21:27:03.521500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.521418 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:27:03.527671 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.526320 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:27:03.530820 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.530777 4926 patch_prober.go:29] interesting pod/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 30 21:27:03.553140 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.553101 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" podUID="76bddd4e-d53f-41f6-95a5-bc445d3eedcf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 30 21:27:03.553771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.540186 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:03.556320 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.556285 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:27:03.558137 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.558096 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" Oct 30 21:27:03.724764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.724724 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:27:03.735382 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.732736 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:27:03.738849 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.735715 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:27:03.749583 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.739521 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" podStartSLOduration=7.558888856 podStartE2EDuration="8.739502541s" totalImagesPullingTime="1.180613685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:02.34105024 +0000 UTC m=+1609.462123985" watchObservedRunningTime="2026-10-30 21:27:03.739502541 +0000 UTC m=+1610.860576288" Oct 30 21:27:03.749583 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.744550 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:27:03.772484 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.772448 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:27:03.968375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.921142 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:27:03.968375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.922604 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:27:03.968375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:03.945197 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:27:04.141132 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:27:04.141023 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc79ad431_18a6_4301_bb75_db4a80b01755.slice/crio-7188e5e7b7b493aea9aa200a49594749614b7f65057f0b6a2918fb647dd066db WatchSource:0}: Error finding container 7188e5e7b7b493aea9aa200a49594749614b7f65057f0b6a2918fb647dd066db: Status 404 returned error can't find the container with id 7188e5e7b7b493aea9aa200a49594749614b7f65057f0b6a2918fb647dd066db Oct 30 21:27:04.144385 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:04.144342 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr"] Oct 30 21:27:04.145275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:04.145203 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:27:04.146758 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:04.146718 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:27:04.148428 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:04.148346 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:27:04.475020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:04.474979 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:27:04.479265 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:04.477858 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:27:04.553539 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:04.553506 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" event={"ID":"c79ad431-18a6-4301-bb75-db4a80b01755","Type":"ContainerStarted","Data":"7188e5e7b7b493aea9aa200a49594749614b7f65057f0b6a2918fb647dd066db"} Oct 30 21:27:05.564547 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:05.564503 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" event={"ID":"c79ad431-18a6-4301-bb75-db4a80b01755","Type":"ContainerStarted","Data":"6a9c97a15f2d1654bf43c88a22a8052f1c25204f0794d0f37e36b3667678be65"} Oct 30 21:27:05.565027 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:05.564595 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" Oct 30 21:27:07.548152 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:27:07.548112 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:27:08.548286 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:08.548255 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" Oct 30 21:27:08.606217 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:08.606153 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" podStartSLOduration=8.264066828 podStartE2EDuration="8.606133099s" totalImagesPullingTime="342.066271ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:05.612419441 +0000 UTC m=+1612.733493188" watchObservedRunningTime="2026-10-30 21:27:08.606133099 +0000 UTC m=+1615.727206846" Oct 30 21:27:11.588176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:11.588141 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" Oct 30 21:27:11.639582 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:11.639539 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:27:15.107402 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:15.107358 4926 scope.go:122] "RemoveContainer" containerID="e60f88117cd80fc420b1915088d493c2182fc92b9f0d7a975c550ef7ccd953f3" Oct 30 21:27:21.046672 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:21.046143 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps"] Oct 30 21:27:21.053013 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:21.052979 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a04cffd9-62zps"] Oct 30 21:27:23.017571 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:23.017533 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4951ebd2-7b73-442d-a1d5-1e517970c885" path="/var/lib/kubelet/pods/4951ebd2-7b73-442d-a1d5-1e517970c885/volumes" Oct 30 21:27:37.489055 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:27:37.488990 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:27:46.150355 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.150315 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz"] Oct 30 21:27:46.150795 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.150530 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" podUID="d02b4c5e-92d2-412b-8073-da97d6fb4a60" containerName="discovery" containerID="cri-o://58883ae13a4f930fb19bd7f7be4cd5e7cb44360b42a925452213072789005a9f" gracePeriod=30 Oct 30 21:27:46.510082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.509608 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:27:46.550647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.550601 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-csr-ca-configmap\") pod \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " Oct 30 21:27:46.550824 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.550672 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-kubeconfig\") pod \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " Oct 30 21:27:46.550890 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.550866 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d02b4c5e-92d2-412b-8073-da97d6fb4a60-cacerts\" (UniqueName: \"kubernetes.io/secret/d02b4c5e-92d2-412b-8073-da97d6fb4a60-cacerts\") pod \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " Oct 30 21:27:46.550961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.550929 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d02b4c5e-92d2-412b-8073-da97d6fb4a60-kube-api-access-rkm77\" (UniqueName: \"kubernetes.io/projected/d02b4c5e-92d2-412b-8073-da97d6fb4a60-kube-api-access-rkm77\") pod \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " Oct 30 21:27:46.551093 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.551073 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-csr-dns-cert\") pod \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " Oct 30 21:27:46.551174 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.551115 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d02b4c5e-92d2-412b-8073-da97d6fb4a60-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d02b4c5e-92d2-412b-8073-da97d6fb4a60-local-certs\") pod \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " Oct 30 21:27:46.551232 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.551219 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-token\" (UniqueName: \"kubernetes.io/projected/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-token\") pod \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\" (UID: \"d02b4c5e-92d2-412b-8073-da97d6fb4a60\") " Oct 30 21:27:46.551423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.551107 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-csr-ca-configmap" pod "d02b4c5e-92d2-412b-8073-da97d6fb4a60" (UID: "d02b4c5e-92d2-412b-8073-da97d6fb4a60"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:27:46.551862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.551833 4926 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-csr-ca-configmap\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:46.553248 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.553222 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-kubeconfig" pod "d02b4c5e-92d2-412b-8073-da97d6fb4a60" (UID: "d02b4c5e-92d2-412b-8073-da97d6fb4a60"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:27:46.553684 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.553630 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02b4c5e-92d2-412b-8073-da97d6fb4a60-kube-api-access-rkm77" pod "d02b4c5e-92d2-412b-8073-da97d6fb4a60" (UID: "d02b4c5e-92d2-412b-8073-da97d6fb4a60"). InnerVolumeSpecName "kube-api-access-rkm77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:27:46.553684 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.553646 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d02b4c5e-92d2-412b-8073-da97d6fb4a60-cacerts" pod "d02b4c5e-92d2-412b-8073-da97d6fb4a60" (UID: "d02b4c5e-92d2-412b-8073-da97d6fb4a60"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:27:46.553916 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.553895 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-token" pod "d02b4c5e-92d2-412b-8073-da97d6fb4a60" (UID: "d02b4c5e-92d2-412b-8073-da97d6fb4a60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:27:46.553994 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.553920 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-csr-dns-cert" pod "d02b4c5e-92d2-412b-8073-da97d6fb4a60" (UID: "d02b4c5e-92d2-412b-8073-da97d6fb4a60"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:27:46.553994 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.553937 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d02b4c5e-92d2-412b-8073-da97d6fb4a60-local-certs" pod "d02b4c5e-92d2-412b-8073-da97d6fb4a60" (UID: "d02b4c5e-92d2-412b-8073-da97d6fb4a60"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:46.653651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.653607 4926 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d02b4c5e-92d2-412b-8073-da97d6fb4a60-cacerts\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:46.653651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.653641 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkm77\" (UniqueName: \"kubernetes.io/projected/d02b4c5e-92d2-412b-8073-da97d6fb4a60-kube-api-access-rkm77\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:46.653651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.653652 4926 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-csr-dns-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:46.653651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.653661 4926 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d02b4c5e-92d2-412b-8073-da97d6fb4a60-local-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:46.653651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.653670 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:46.654161 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.653678 4926 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d02b4c5e-92d2-412b-8073-da97d6fb4a60-istio-kubeconfig\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:46.730256 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.730156 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw"] Oct 30 21:27:46.730445 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:46.730374 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" podUID="2faae7ee-cee5-49cc-a4e2-22adc43c8289" containerName="istio-proxy" containerID="cri-o://e88570d94185fb9bf8d1edb61636c843f30baa828e2eaad1976e05d1c555f0fc" gracePeriod=30 Oct 30 21:27:47.084054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.083846 4926 generic.go:356] "Generic (PLEG): container finished" podID="d02b4c5e-92d2-412b-8073-da97d6fb4a60" containerID="58883ae13a4f930fb19bd7f7be4cd5e7cb44360b42a925452213072789005a9f" exitCode=0 Oct 30 21:27:47.084267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.084082 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" event={"ID":"d02b4c5e-92d2-412b-8073-da97d6fb4a60","Type":"ContainerDied","Data":"58883ae13a4f930fb19bd7f7be4cd5e7cb44360b42a925452213072789005a9f"} Oct 30 21:27:47.084267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.084136 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" event={"ID":"d02b4c5e-92d2-412b-8073-da97d6fb4a60","Type":"ContainerDied","Data":"5300bd5526987c91ec915b73434bee6791fcba6130f592cd7d48b844ccbe3251"} Oct 30 21:27:47.084267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.084163 4926 scope.go:122] "RemoveContainer" containerID="58883ae13a4f930fb19bd7f7be4cd5e7cb44360b42a925452213072789005a9f" Oct 30 21:27:47.084449 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.084338 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz" Oct 30 21:27:47.193519 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.193213 4926 scope.go:122] "RemoveContainer" containerID="58883ae13a4f930fb19bd7f7be4cd5e7cb44360b42a925452213072789005a9f" Oct 30 21:27:47.193852 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:27:47.193629 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58883ae13a4f930fb19bd7f7be4cd5e7cb44360b42a925452213072789005a9f\": container with ID starting with 58883ae13a4f930fb19bd7f7be4cd5e7cb44360b42a925452213072789005a9f not found: ID does not exist" containerID="58883ae13a4f930fb19bd7f7be4cd5e7cb44360b42a925452213072789005a9f" Oct 30 21:27:47.193852 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.193669 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58883ae13a4f930fb19bd7f7be4cd5e7cb44360b42a925452213072789005a9f"} err="failed to get container status \"58883ae13a4f930fb19bd7f7be4cd5e7cb44360b42a925452213072789005a9f\": rpc error: code = NotFound desc = could not find container \"58883ae13a4f930fb19bd7f7be4cd5e7cb44360b42a925452213072789005a9f\": container with ID starting with 58883ae13a4f930fb19bd7f7be4cd5e7cb44360b42a925452213072789005a9f not found: ID does not exist" Oct 30 21:27:47.238060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.232095 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz"] Oct 30 21:27:47.270937 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.256182 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-hd6sz"] Oct 30 21:27:47.547732 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.547691 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x"] Oct 30 21:27:47.548821 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.548797 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d02b4c5e-92d2-412b-8073-da97d6fb4a60" containerName="discovery" Oct 30 21:27:47.548896 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.548825 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d02b4c5e-92d2-412b-8073-da97d6fb4a60" containerName="discovery" Oct 30 21:27:47.549070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.549057 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d02b4c5e-92d2-412b-8073-da97d6fb4a60" containerName="discovery" Oct 30 21:27:47.554755 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.554731 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.577615 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.577583 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c9477c-openshift-default-dockercfg-8hhn7\"" Oct 30 21:27:47.610062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.609758 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-credential-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.610062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.609870 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-workload-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.610062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.609947 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-workload-certs\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.610062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.610017 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istiod-ca-cert\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.610437 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.610154 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-envoy\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.614058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.611217 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-token\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.614058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.611323 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g64fr\" (UniqueName: \"kubernetes.io/projected/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-kube-api-access-g64fr\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.614058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.611409 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-data\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.614058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.611438 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-podinfo\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.718816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.713116 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-token\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.718816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.713212 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g64fr\" (UniqueName: \"kubernetes.io/projected/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-kube-api-access-g64fr\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.718816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.713288 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-data\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.718816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.713314 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-podinfo\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.718816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.713363 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-credential-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.718816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.713436 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-workload-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.718816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.713497 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-workload-certs\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.718816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.713563 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istiod-ca-cert\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.718816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.713668 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-envoy\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.718816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.714550 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-workload-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.718816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.714803 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-credential-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.718816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.715582 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-data\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.718816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.715843 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-workload-certs\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.718816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.717368 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-envoy\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.726759 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.726725 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istiod-ca-cert\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.730582 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.730543 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-podinfo\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.744103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.743986 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g64fr\" (UniqueName: \"kubernetes.io/projected/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-kube-api-access-g64fr\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.744279 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.744118 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-token\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:47.776820 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.776780 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x"] Oct 30 21:27:47.872701 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:47.872602 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:48.153140 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:48.153001 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n"] Oct 30 21:27:48.153334 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:48.153224 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" podUID="94eead01-fd4c-4c47-8d70-527ba6643cd5" containerName="istio-proxy" containerID="cri-o://fa203018dbac282594a360fb682cc11b3fc41f5488fb63f241ebec303b46bc95" gracePeriod=30 Oct 30 21:27:48.324643 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:48.324599 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:27:48.325160 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:48.324701 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:27:48.325160 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:48.324750 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:27:48.339429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:48.339391 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x"] Oct 30 21:27:48.437740 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:48.437710 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2"] Oct 30 21:27:48.438165 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:48.438135 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" podUID="94ddad94-5958-43f7-b1fc-4c66a3117a29" containerName="istio-proxy" containerID="cri-o://a98ca3aa4acb654926715aafa87caffc7d3e4092ede54a439fa78e509d778072" gracePeriod=30 Oct 30 21:27:49.093056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:49.069824 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d02b4c5e-92d2-412b-8073-da97d6fb4a60" path="/var/lib/kubelet/pods/d02b4c5e-92d2-412b-8073-da97d6fb4a60/volumes" Oct 30 21:27:49.234066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:49.201062 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" event={"ID":"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7","Type":"ContainerStarted","Data":"fc2716b6916caa2faaea4f9e113d0f21dfd630c4d58ffdf63aac04fd2b32b929"} Oct 30 21:27:49.234066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:49.201134 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" event={"ID":"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7","Type":"ContainerStarted","Data":"37a5ec3ba8979b71986d592b915e4639291fff78b7a68950878024b0a15a86ec"} Oct 30 21:27:49.873375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:49.873335 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:49.880727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:49.880679 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:49.945886 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:49.945829 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" podStartSLOduration=2.94581458 podStartE2EDuration="2.94581458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:49.289408007 +0000 UTC m=+1656.410481753" watchObservedRunningTime="2026-10-30 21:27:49.94581458 +0000 UTC m=+1657.066888326" Oct 30 21:27:50.212114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:50.212069 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:50.213638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:50.213607 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:27:52.048774 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.048739 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:27:52.144897 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.144857 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-token\" (UniqueName: \"kubernetes.io/projected/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-token\") pod \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " Oct 30 21:27:52.145145 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.145007 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-podinfo\") pod \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " Oct 30 21:27:52.145145 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.145064 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-workload-certs\") pod \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " Oct 30 21:27:52.145145 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.145127 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-credential-socket\") pod \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " Oct 30 21:27:52.145339 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.145208 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2faae7ee-cee5-49cc-a4e2-22adc43c8289-kube-api-access-mxvv7\" (UniqueName: \"kubernetes.io/projected/2faae7ee-cee5-49cc-a4e2-22adc43c8289-kube-api-access-mxvv7\") pod \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " Oct 30 21:27:52.145339 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.145238 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-workload-socket\") pod \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " Oct 30 21:27:52.145339 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.145298 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-data\") pod \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " Oct 30 21:27:52.145561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.145517 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-credential-socket" pod "2faae7ee-cee5-49cc-a4e2-22adc43c8289" (UID: "2faae7ee-cee5-49cc-a4e2-22adc43c8289"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:52.145630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.145591 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-data" pod "2faae7ee-cee5-49cc-a4e2-22adc43c8289" (UID: "2faae7ee-cee5-49cc-a4e2-22adc43c8289"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:52.145752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.145735 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-workload-certs" pod "2faae7ee-cee5-49cc-a4e2-22adc43c8289" (UID: "2faae7ee-cee5-49cc-a4e2-22adc43c8289"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:52.145843 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.145822 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-workload-socket" pod "2faae7ee-cee5-49cc-a4e2-22adc43c8289" (UID: "2faae7ee-cee5-49cc-a4e2-22adc43c8289"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:52.145893 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.145864 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-envoy\") pod \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " Oct 30 21:27:52.146142 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.146092 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istiod-ca-cert\") pod \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\" (UID: \"2faae7ee-cee5-49cc-a4e2-22adc43c8289\") " Oct 30 21:27:52.146501 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.146449 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istiod-ca-cert" pod "2faae7ee-cee5-49cc-a4e2-22adc43c8289" (UID: "2faae7ee-cee5-49cc-a4e2-22adc43c8289"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:27:52.146999 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.146975 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:52.147127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.147003 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:52.147127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.147019 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:52.147127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.147102 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:52.147127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.147119 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:52.147718 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.147687 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-token" pod "2faae7ee-cee5-49cc-a4e2-22adc43c8289" (UID: "2faae7ee-cee5-49cc-a4e2-22adc43c8289"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:27:52.148664 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.148635 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-podinfo" pod "2faae7ee-cee5-49cc-a4e2-22adc43c8289" (UID: "2faae7ee-cee5-49cc-a4e2-22adc43c8289"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:27:52.148772 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.148735 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-envoy" pod "2faae7ee-cee5-49cc-a4e2-22adc43c8289" (UID: "2faae7ee-cee5-49cc-a4e2-22adc43c8289"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:52.152072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.152047 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2faae7ee-cee5-49cc-a4e2-22adc43c8289-kube-api-access-mxvv7" pod "2faae7ee-cee5-49cc-a4e2-22adc43c8289" (UID: "2faae7ee-cee5-49cc-a4e2-22adc43c8289"). InnerVolumeSpecName "kube-api-access-mxvv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:27:52.231903 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.231863 4926 generic.go:356] "Generic (PLEG): container finished" podID="2faae7ee-cee5-49cc-a4e2-22adc43c8289" containerID="e88570d94185fb9bf8d1edb61636c843f30baa828e2eaad1976e05d1c555f0fc" exitCode=0 Oct 30 21:27:52.232094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.231943 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" Oct 30 21:27:52.232094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.231946 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" event={"ID":"2faae7ee-cee5-49cc-a4e2-22adc43c8289","Type":"ContainerDied","Data":"e88570d94185fb9bf8d1edb61636c843f30baa828e2eaad1976e05d1c555f0fc"} Oct 30 21:27:52.232094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.232071 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw" event={"ID":"2faae7ee-cee5-49cc-a4e2-22adc43c8289","Type":"ContainerDied","Data":"0480e2dc8d8f324599ec615de1910c45dda58440e8a20f01f36d5bfbb27a6c8e"} Oct 30 21:27:52.232209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.232107 4926 scope.go:122] "RemoveContainer" containerID="e88570d94185fb9bf8d1edb61636c843f30baa828e2eaad1976e05d1c555f0fc" Oct 30 21:27:52.248993 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.248959 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:52.248993 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.248995 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxvv7\" (UniqueName: \"kubernetes.io/projected/2faae7ee-cee5-49cc-a4e2-22adc43c8289-kube-api-access-mxvv7\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:52.249359 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.249011 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:52.249359 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.249026 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2faae7ee-cee5-49cc-a4e2-22adc43c8289-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:52.251805 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.251783 4926 scope.go:122] "RemoveContainer" containerID="e88570d94185fb9bf8d1edb61636c843f30baa828e2eaad1976e05d1c555f0fc" Oct 30 21:27:52.252343 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:27:52.252223 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e88570d94185fb9bf8d1edb61636c843f30baa828e2eaad1976e05d1c555f0fc\": container with ID starting with e88570d94185fb9bf8d1edb61636c843f30baa828e2eaad1976e05d1c555f0fc not found: ID does not exist" containerID="e88570d94185fb9bf8d1edb61636c843f30baa828e2eaad1976e05d1c555f0fc" Oct 30 21:27:52.252343 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.252262 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88570d94185fb9bf8d1edb61636c843f30baa828e2eaad1976e05d1c555f0fc"} err="failed to get container status \"e88570d94185fb9bf8d1edb61636c843f30baa828e2eaad1976e05d1c555f0fc\": rpc error: code = NotFound desc = could not find container \"e88570d94185fb9bf8d1edb61636c843f30baa828e2eaad1976e05d1c555f0fc\": container with ID starting with e88570d94185fb9bf8d1edb61636c843f30baa828e2eaad1976e05d1c555f0fc not found: ID does not exist" Oct 30 21:27:52.342114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.338199 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw"] Oct 30 21:27:52.373074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:52.370063 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-qc4rw"] Oct 30 21:27:53.018083 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.018048 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2faae7ee-cee5-49cc-a4e2-22adc43c8289" path="/var/lib/kubelet/pods/2faae7ee-cee5-49cc-a4e2-22adc43c8289/volumes" Oct 30 21:27:53.736373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.736342 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:27:53.743706 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.743656 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94eead01-fd4c-4c47-8d70-527ba6643cd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94eead01-fd4c-4c47-8d70-527ba6643cd5-istiod-ca-cert\") pod \"94eead01-fd4c-4c47-8d70-527ba6643cd5\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " Oct 30 21:27:53.744309 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.744277 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94eead01-fd4c-4c47-8d70-527ba6643cd5-istiod-ca-cert" pod "94eead01-fd4c-4c47-8d70-527ba6643cd5" (UID: "94eead01-fd4c-4c47-8d70-527ba6643cd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:27:53.745405 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.745373 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94eead01-fd4c-4c47-8d70-527ba6643cd5-kube-api-access-g8jw5\" (UniqueName: \"kubernetes.io/projected/94eead01-fd4c-4c47-8d70-527ba6643cd5-kube-api-access-g8jw5\") pod \"94eead01-fd4c-4c47-8d70-527ba6643cd5\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " Oct 30 21:27:53.745531 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.745449 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-podinfo\") pod \"94eead01-fd4c-4c47-8d70-527ba6643cd5\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " Oct 30 21:27:53.745531 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.745510 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-envoy\") pod \"94eead01-fd4c-4c47-8d70-527ba6643cd5\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " Oct 30 21:27:53.745648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.745540 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-credential-socket\") pod \"94eead01-fd4c-4c47-8d70-527ba6643cd5\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " Oct 30 21:27:53.943157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.817158 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-credential-socket" pod "94eead01-fd4c-4c47-8d70-527ba6643cd5" (UID: "94eead01-fd4c-4c47-8d70-527ba6643cd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:53.943157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.817454 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-workload-certs" pod "94eead01-fd4c-4c47-8d70-527ba6643cd5" (UID: "94eead01-fd4c-4c47-8d70-527ba6643cd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:53.943157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.817510 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-workload-certs\") pod \"94eead01-fd4c-4c47-8d70-527ba6643cd5\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " Oct 30 21:27:53.943157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.817580 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-data\") pod \"94eead01-fd4c-4c47-8d70-527ba6643cd5\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " Oct 30 21:27:53.943157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.817626 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-token\" (UniqueName: \"kubernetes.io/projected/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-token\") pod \"94eead01-fd4c-4c47-8d70-527ba6643cd5\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " Oct 30 21:27:53.943157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.817695 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-workload-socket\") pod \"94eead01-fd4c-4c47-8d70-527ba6643cd5\" (UID: \"94eead01-fd4c-4c47-8d70-527ba6643cd5\") " Oct 30 21:27:53.943157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.818585 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94eead01-fd4c-4c47-8d70-527ba6643cd5-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:53.943157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.818609 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:53.943157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.818624 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:53.943157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.820428 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-workload-socket" pod "94eead01-fd4c-4c47-8d70-527ba6643cd5" (UID: "94eead01-fd4c-4c47-8d70-527ba6643cd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:53.943157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.821256 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-data" pod "94eead01-fd4c-4c47-8d70-527ba6643cd5" (UID: "94eead01-fd4c-4c47-8d70-527ba6643cd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:53.943157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.920249 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:53.943157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.920280 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:53.955078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.951917 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94eead01-fd4c-4c47-8d70-527ba6643cd5-kube-api-access-g8jw5" pod "94eead01-fd4c-4c47-8d70-527ba6643cd5" (UID: "94eead01-fd4c-4c47-8d70-527ba6643cd5"). InnerVolumeSpecName "kube-api-access-g8jw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:27:53.989058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:53.986242 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-envoy" pod "94eead01-fd4c-4c47-8d70-527ba6643cd5" (UID: "94eead01-fd4c-4c47-8d70-527ba6643cd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:54.032221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.023666 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8jw5\" (UniqueName: \"kubernetes.io/projected/94eead01-fd4c-4c47-8d70-527ba6643cd5-kube-api-access-g8jw5\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:54.032221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.023698 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:54.032221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.027477 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-podinfo" pod "94eead01-fd4c-4c47-8d70-527ba6643cd5" (UID: "94eead01-fd4c-4c47-8d70-527ba6643cd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:27:54.091053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.079546 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-token" pod "94eead01-fd4c-4c47-8d70-527ba6643cd5" (UID: "94eead01-fd4c-4c47-8d70-527ba6643cd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:27:54.137084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.127534 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:54.137084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.127579 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94eead01-fd4c-4c47-8d70-527ba6643cd5-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:54.242534 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.242505 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:27:54.267647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.267613 4926 generic.go:356] "Generic (PLEG): container finished" podID="94eead01-fd4c-4c47-8d70-527ba6643cd5" containerID="fa203018dbac282594a360fb682cc11b3fc41f5488fb63f241ebec303b46bc95" exitCode=0 Oct 30 21:27:54.267855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.267658 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" Oct 30 21:27:54.267855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.267702 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" event={"ID":"94eead01-fd4c-4c47-8d70-527ba6643cd5","Type":"ContainerDied","Data":"fa203018dbac282594a360fb682cc11b3fc41f5488fb63f241ebec303b46bc95"} Oct 30 21:27:54.267855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.267780 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n" event={"ID":"94eead01-fd4c-4c47-8d70-527ba6643cd5","Type":"ContainerDied","Data":"d4a1dc274a3033f63b354f5210b4c7dc44aa0ba72b5be99c97a5c5b8ce95294a"} Oct 30 21:27:54.267855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.267811 4926 scope.go:122] "RemoveContainer" containerID="fa203018dbac282594a360fb682cc11b3fc41f5488fb63f241ebec303b46bc95" Oct 30 21:27:54.269937 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.269901 4926 generic.go:356] "Generic (PLEG): container finished" podID="94ddad94-5958-43f7-b1fc-4c66a3117a29" containerID="a98ca3aa4acb654926715aafa87caffc7d3e4092ede54a439fa78e509d778072" exitCode=0 Oct 30 21:27:54.270278 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.269979 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" Oct 30 21:27:54.270278 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.269988 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" event={"ID":"94ddad94-5958-43f7-b1fc-4c66a3117a29","Type":"ContainerDied","Data":"a98ca3aa4acb654926715aafa87caffc7d3e4092ede54a439fa78e509d778072"} Oct 30 21:27:54.270278 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.270253 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2" event={"ID":"94ddad94-5958-43f7-b1fc-4c66a3117a29","Type":"ContainerDied","Data":"946ccd877e9c9c4412aa216273577dc781cd9f000bdd30e795fd30c8f7fa841d"} Oct 30 21:27:54.280737 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.280712 4926 scope.go:122] "RemoveContainer" containerID="fa203018dbac282594a360fb682cc11b3fc41f5488fb63f241ebec303b46bc95" Oct 30 21:27:54.281111 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:27:54.281086 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa203018dbac282594a360fb682cc11b3fc41f5488fb63f241ebec303b46bc95\": container with ID starting with fa203018dbac282594a360fb682cc11b3fc41f5488fb63f241ebec303b46bc95 not found: ID does not exist" containerID="fa203018dbac282594a360fb682cc11b3fc41f5488fb63f241ebec303b46bc95" Oct 30 21:27:54.281207 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.281124 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa203018dbac282594a360fb682cc11b3fc41f5488fb63f241ebec303b46bc95"} err="failed to get container status \"fa203018dbac282594a360fb682cc11b3fc41f5488fb63f241ebec303b46bc95\": rpc error: code = NotFound desc = could not find container \"fa203018dbac282594a360fb682cc11b3fc41f5488fb63f241ebec303b46bc95\": container with ID starting with fa203018dbac282594a360fb682cc11b3fc41f5488fb63f241ebec303b46bc95 not found: ID does not exist" Oct 30 21:27:54.281207 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.281149 4926 scope.go:122] "RemoveContainer" containerID="a98ca3aa4acb654926715aafa87caffc7d3e4092ede54a439fa78e509d778072" Oct 30 21:27:54.294752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.294620 4926 scope.go:122] "RemoveContainer" containerID="a98ca3aa4acb654926715aafa87caffc7d3e4092ede54a439fa78e509d778072" Oct 30 21:27:54.295092 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:27:54.295055 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a98ca3aa4acb654926715aafa87caffc7d3e4092ede54a439fa78e509d778072\": container with ID starting with a98ca3aa4acb654926715aafa87caffc7d3e4092ede54a439fa78e509d778072 not found: ID does not exist" containerID="a98ca3aa4acb654926715aafa87caffc7d3e4092ede54a439fa78e509d778072" Oct 30 21:27:54.295193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.295107 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a98ca3aa4acb654926715aafa87caffc7d3e4092ede54a439fa78e509d778072"} err="failed to get container status \"a98ca3aa4acb654926715aafa87caffc7d3e4092ede54a439fa78e509d778072\": rpc error: code = NotFound desc = could not find container \"a98ca3aa4acb654926715aafa87caffc7d3e4092ede54a439fa78e509d778072\": container with ID starting with a98ca3aa4acb654926715aafa87caffc7d3e4092ede54a439fa78e509d778072 not found: ID does not exist" Oct 30 21:27:54.330419 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.330379 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-token\" (UniqueName: \"kubernetes.io/projected/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-token\") pod \"94ddad94-5958-43f7-b1fc-4c66a3117a29\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " Oct 30 21:27:54.330638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.330448 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-workload-socket\") pod \"94ddad94-5958-43f7-b1fc-4c66a3117a29\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " Oct 30 21:27:54.330638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.330517 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94ddad94-5958-43f7-b1fc-4c66a3117a29-kube-api-access-2mvf5\" (UniqueName: \"kubernetes.io/projected/94ddad94-5958-43f7-b1fc-4c66a3117a29-kube-api-access-2mvf5\") pod \"94ddad94-5958-43f7-b1fc-4c66a3117a29\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " Oct 30 21:27:54.330638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.330557 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94ddad94-5958-43f7-b1fc-4c66a3117a29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ddad94-5958-43f7-b1fc-4c66a3117a29-istiod-ca-cert\") pod \"94ddad94-5958-43f7-b1fc-4c66a3117a29\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " Oct 30 21:27:54.330817 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.330673 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-credential-socket\") pod \"94ddad94-5958-43f7-b1fc-4c66a3117a29\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " Oct 30 21:27:54.330817 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.330718 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-envoy\") pod \"94ddad94-5958-43f7-b1fc-4c66a3117a29\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " Oct 30 21:27:54.330817 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.330811 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-podinfo\") pod \"94ddad94-5958-43f7-b1fc-4c66a3117a29\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " Oct 30 21:27:54.330975 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.330840 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-workload-certs\") pod \"94ddad94-5958-43f7-b1fc-4c66a3117a29\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " Oct 30 21:27:54.330975 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.330897 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-data\") pod \"94ddad94-5958-43f7-b1fc-4c66a3117a29\" (UID: \"94ddad94-5958-43f7-b1fc-4c66a3117a29\") " Oct 30 21:27:54.331199 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.331155 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-workload-socket" pod "94ddad94-5958-43f7-b1fc-4c66a3117a29" (UID: "94ddad94-5958-43f7-b1fc-4c66a3117a29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:54.331295 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.331240 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ddad94-5958-43f7-b1fc-4c66a3117a29-istiod-ca-cert" pod "94ddad94-5958-43f7-b1fc-4c66a3117a29" (UID: "94ddad94-5958-43f7-b1fc-4c66a3117a29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:27:54.331517 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.331480 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-data" pod "94ddad94-5958-43f7-b1fc-4c66a3117a29" (UID: "94ddad94-5958-43f7-b1fc-4c66a3117a29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:54.331876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.331850 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-workload-certs" pod "94ddad94-5958-43f7-b1fc-4c66a3117a29" (UID: "94ddad94-5958-43f7-b1fc-4c66a3117a29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:54.331960 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.331908 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:54.331960 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.331931 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:54.331960 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.331944 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94ddad94-5958-43f7-b1fc-4c66a3117a29-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:54.332175 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.332150 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-credential-socket" pod "94ddad94-5958-43f7-b1fc-4c66a3117a29" (UID: "94ddad94-5958-43f7-b1fc-4c66a3117a29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:54.333216 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.333184 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-token" pod "94ddad94-5958-43f7-b1fc-4c66a3117a29" (UID: "94ddad94-5958-43f7-b1fc-4c66a3117a29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:27:54.333960 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.333938 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-envoy" pod "94ddad94-5958-43f7-b1fc-4c66a3117a29" (UID: "94ddad94-5958-43f7-b1fc-4c66a3117a29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:27:54.334373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.334345 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-podinfo" pod "94ddad94-5958-43f7-b1fc-4c66a3117a29" (UID: "94ddad94-5958-43f7-b1fc-4c66a3117a29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:27:54.334462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.334408 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94ddad94-5958-43f7-b1fc-4c66a3117a29-kube-api-access-2mvf5" pod "94ddad94-5958-43f7-b1fc-4c66a3117a29" (UID: "94ddad94-5958-43f7-b1fc-4c66a3117a29"). InnerVolumeSpecName "kube-api-access-2mvf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:27:54.434070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.433997 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:54.434070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.434068 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:54.434399 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.434085 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:54.434399 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.434097 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94ddad94-5958-43f7-b1fc-4c66a3117a29-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:54.434399 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.434108 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94ddad94-5958-43f7-b1fc-4c66a3117a29-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:54.434399 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.434120 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mvf5\" (UniqueName: \"kubernetes.io/projected/94ddad94-5958-43f7-b1fc-4c66a3117a29-kube-api-access-2mvf5\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:27:54.438875 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.438844 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n"] Oct 30 21:27:54.454511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.454415 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss"] Oct 30 21:27:54.455429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.455392 4926 cpu_manager.go:414] "containerMap: removing container" podUID="2faae7ee-cee5-49cc-a4e2-22adc43c8289" containerName="istio-proxy" Oct 30 21:27:54.455429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.455429 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2faae7ee-cee5-49cc-a4e2-22adc43c8289" containerName="istio-proxy" Oct 30 21:27:54.455650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.455474 4926 cpu_manager.go:414] "containerMap: removing container" podUID="94ddad94-5958-43f7-b1fc-4c66a3117a29" containerName="istio-proxy" Oct 30 21:27:54.455650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.455483 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94ddad94-5958-43f7-b1fc-4c66a3117a29" containerName="istio-proxy" Oct 30 21:27:54.455650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.455550 4926 cpu_manager.go:414] "containerMap: removing container" podUID="94eead01-fd4c-4c47-8d70-527ba6643cd5" containerName="istio-proxy" Oct 30 21:27:54.455650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.455560 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94eead01-fd4c-4c47-8d70-527ba6643cd5" containerName="istio-proxy" Oct 30 21:27:54.455856 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.455710 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="2faae7ee-cee5-49cc-a4e2-22adc43c8289" containerName="istio-proxy" Oct 30 21:27:54.455856 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.455733 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="94ddad94-5958-43f7-b1fc-4c66a3117a29" containerName="istio-proxy" Oct 30 21:27:54.455856 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.455753 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="94eead01-fd4c-4c47-8d70-527ba6643cd5" containerName="istio-proxy" Oct 30 21:27:54.461493 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.461469 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.488483 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.488451 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-gcq8n"] Oct 30 21:27:54.502397 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.502361 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-200d11-openshift-default-dockercfg-zhzhk\"" Oct 30 21:27:54.506553 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.506517 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss"] Oct 30 21:27:54.535614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.535579 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk67j\" (UniqueName: \"kubernetes.io/projected/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-kube-api-access-tk67j\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.535792 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.535625 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-credential-socket\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.535792 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.535763 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istiod-ca-cert\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.535898 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.535878 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-data\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.535957 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.535943 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-workload-socket\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.535990 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.535974 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-workload-certs\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.536096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.536077 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-envoy\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.536166 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.536150 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-token\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.536318 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.536296 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-podinfo\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.638283 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.638241 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-token\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.638485 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.638411 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-podinfo\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.638562 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.638504 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk67j\" (UniqueName: \"kubernetes.io/projected/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-kube-api-access-tk67j\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.638562 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.638548 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-credential-socket\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.638686 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.638615 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istiod-ca-cert\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.638743 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.638699 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-data\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.638795 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.638750 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-workload-socket\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.638795 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.638778 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-workload-certs\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.638892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.638829 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-envoy\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.643879 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.643848 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-data\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.644305 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.644275 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istiod-ca-cert\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.644481 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.644461 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-workload-socket\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.644619 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.644597 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-workload-certs\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.645056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.639711 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-credential-socket\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.646187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.646168 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-envoy\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.648416 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.648389 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-podinfo\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.696818 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.688294 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-token\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.696818 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.694207 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk67j\" (UniqueName: \"kubernetes.io/projected/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-kube-api-access-tk67j\") pod \"e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.755181 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.754143 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2"] Oct 30 21:27:54.783375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.783331 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:54.792026 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:54.786280 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-fmch2"] Oct 30 21:27:55.032216 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.031218 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94ddad94-5958-43f7-b1fc-4c66a3117a29" path="/var/lib/kubelet/pods/94ddad94-5958-43f7-b1fc-4c66a3117a29/volumes" Oct 30 21:27:55.032216 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.031797 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94eead01-fd4c-4c47-8d70-527ba6643cd5" path="/var/lib/kubelet/pods/94eead01-fd4c-4c47-8d70-527ba6643cd5/volumes" Oct 30 21:27:55.234413 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.234143 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss"] Oct 30 21:27:55.241071 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.241021 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:27:55.241189 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.241106 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:27:55.241189 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.241137 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:27:55.359821 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.359782 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" event={"ID":"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d","Type":"ContainerStarted","Data":"8bc7539c8dfd02e99d2a596f7e759fb952540ad6c51a95a26f26a0fe9f7f5e62"} Oct 30 21:27:55.466008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.465967 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7"] Oct 30 21:27:55.514795 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.508456 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7" Oct 30 21:27:55.519239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.515198 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7"] Oct 30 21:27:55.531775 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.531721 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c9477c-dockercfg-x2wxt\"" Oct 30 21:27:55.533497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.533457 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c9477c\"" Oct 30 21:27:55.544780 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.544185 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk"] Oct 30 21:27:55.563114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.551496 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk" Oct 30 21:27:55.603107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.597493 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ff5943-9467-4ce9-a354-9070c9cef92f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk\" (UID: \"91ff5943-9467-4ce9-a354-9070c9cef92f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk" Oct 30 21:27:55.603107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.597570 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd180d75-5253-4485-8eed-d47bff0287b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7\" (UID: \"cd180d75-5253-4485-8eed-d47bff0287b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7" Oct 30 21:27:55.603107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.597708 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dr27\" (UniqueName: \"kubernetes.io/projected/91ff5943-9467-4ce9-a354-9070c9cef92f-kube-api-access-5dr27\") pod \"payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk\" (UID: \"91ff5943-9467-4ce9-a354-9070c9cef92f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk" Oct 30 21:27:55.603107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.597743 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d96c2\" (UniqueName: \"kubernetes.io/projected/cd180d75-5253-4485-8eed-d47bff0287b2-kube-api-access-d96c2\") pod \"payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7\" (UID: \"cd180d75-5253-4485-8eed-d47bff0287b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7" Oct 30 21:27:55.736809 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.702632 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ff5943-9467-4ce9-a354-9070c9cef92f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk\" (UID: \"91ff5943-9467-4ce9-a354-9070c9cef92f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk" Oct 30 21:27:55.736809 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.702703 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd180d75-5253-4485-8eed-d47bff0287b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7\" (UID: \"cd180d75-5253-4485-8eed-d47bff0287b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7" Oct 30 21:27:55.736809 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.702839 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dr27\" (UniqueName: \"kubernetes.io/projected/91ff5943-9467-4ce9-a354-9070c9cef92f-kube-api-access-5dr27\") pod \"payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk\" (UID: \"91ff5943-9467-4ce9-a354-9070c9cef92f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk" Oct 30 21:27:55.736809 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.702872 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d96c2\" (UniqueName: \"kubernetes.io/projected/cd180d75-5253-4485-8eed-d47bff0287b2-kube-api-access-d96c2\") pod \"payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7\" (UID: \"cd180d75-5253-4485-8eed-d47bff0287b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7" Oct 30 21:27:55.736809 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.715301 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd180d75-5253-4485-8eed-d47bff0287b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7\" (UID: \"cd180d75-5253-4485-8eed-d47bff0287b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7" Oct 30 21:27:55.736809 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.735500 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ff5943-9467-4ce9-a354-9070c9cef92f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk\" (UID: \"91ff5943-9467-4ce9-a354-9070c9cef92f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk" Oct 30 21:27:55.786130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.761698 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk"] Oct 30 21:27:55.855110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.850864 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dr27\" (UniqueName: \"kubernetes.io/projected/91ff5943-9467-4ce9-a354-9070c9cef92f-kube-api-access-5dr27\") pod \"payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk\" (UID: \"91ff5943-9467-4ce9-a354-9070c9cef92f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk" Oct 30 21:27:55.982990 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:55.982959 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d96c2\" (UniqueName: \"kubernetes.io/projected/cd180d75-5253-4485-8eed-d47bff0287b2-kube-api-access-d96c2\") pod \"payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7\" (UID: \"cd180d75-5253-4485-8eed-d47bff0287b2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7" Oct 30 21:27:56.010581 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:56.004162 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk" Oct 30 21:27:56.132476 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:56.132185 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7" Oct 30 21:27:56.415266 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:56.414194 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" event={"ID":"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d","Type":"ContainerStarted","Data":"62d4396fc9cec9e296628ccfc5a668a4237aea8fab4eacf7bc264d64fe6d9a4e"} Oct 30 21:27:56.418088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:56.418059 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk"] Oct 30 21:27:56.420019 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:27:56.419969 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ff5943_9467_4ce9_a354_9070c9cef92f.slice/crio-cd1b6bb6309a2bf26f9e7ee3e72ac75701df7f5d78bafc8cead2f0868f3c0de7 WatchSource:0}: Error finding container cd1b6bb6309a2bf26f9e7ee3e72ac75701df7f5d78bafc8cead2f0868f3c0de7: Status 404 returned error can't find the container with id cd1b6bb6309a2bf26f9e7ee3e72ac75701df7f5d78bafc8cead2f0868f3c0de7 Oct 30 21:27:56.514781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:56.514714 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" podStartSLOduration=2.514680793 podStartE2EDuration="2.514680793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:56.474530143 +0000 UTC m=+1663.595603890" watchObservedRunningTime="2026-10-30 21:27:56.514680793 +0000 UTC m=+1663.635754542" Oct 30 21:27:56.652591 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:56.652539 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7"] Oct 30 21:27:56.791550 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:56.791513 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:56.805941 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:56.805606 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:57.091113 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:57.090084 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw"] Oct 30 21:27:57.106592 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:57.106551 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a814527b-sbwlw"] Oct 30 21:27:57.425985 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:57.425943 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk" event={"ID":"91ff5943-9467-4ce9-a354-9070c9cef92f","Type":"ContainerStarted","Data":"4d046422c4dc1acaff79f493b824bc152f48e874058e79879c30d9f8357d5116"} Oct 30 21:27:57.427095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:57.427065 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk" Oct 30 21:27:57.427275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:57.427110 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk" event={"ID":"91ff5943-9467-4ce9-a354-9070c9cef92f","Type":"ContainerStarted","Data":"cd1b6bb6309a2bf26f9e7ee3e72ac75701df7f5d78bafc8cead2f0868f3c0de7"} Oct 30 21:27:57.428824 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:57.428796 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7" event={"ID":"cd180d75-5253-4485-8eed-d47bff0287b2","Type":"ContainerStarted","Data":"ab04111f4cfd5d5f5331394205c7ef8903303aaaf226d326ebbcfa38fe265d60"} Oct 30 21:27:57.428943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:57.428842 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7" event={"ID":"cd180d75-5253-4485-8eed-d47bff0287b2","Type":"ContainerStarted","Data":"0ce110f58ab86a758d88602bee710cd06e6f943808c9a18b04fcf8a359ffe229"} Oct 30 21:27:57.428943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:57.428868 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7" Oct 30 21:27:57.429234 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:57.429214 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:57.430096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:57.430077 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:27:57.509058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:57.503015 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk" podStartSLOduration=2.502992757 podStartE2EDuration="2.502992757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:57.492487653 +0000 UTC m=+1664.613561401" watchObservedRunningTime="2026-10-30 21:27:57.502992757 +0000 UTC m=+1664.624066504" Oct 30 21:27:57.621054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:57.613843 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7" podStartSLOduration=2.613817257 podStartE2EDuration="2.613817257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:27:57.558679834 +0000 UTC m=+1664.679753582" watchObservedRunningTime="2026-10-30 21:27:57.613817257 +0000 UTC m=+1664.734891004" Oct 30 21:27:59.021468 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:27:59.021428 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8da07db-83fb-49ad-950c-9206d47c0bec" path="/var/lib/kubelet/pods/f8da07db-83fb-49ad-950c-9206d47c0bec/volumes" Oct 30 21:28:01.358597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:01.358557 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk"] Oct 30 21:28:01.364580 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:01.364552 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk" Oct 30 21:28:01.384465 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:01.384421 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-200d11\"" Oct 30 21:28:01.391163 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:01.391131 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk"] Oct 30 21:28:01.394294 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:01.394198 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-200d11-dockercfg-rm499\"" Oct 30 21:28:01.404507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:01.404474 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj"] Oct 30 21:28:01.410047 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:01.410008 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj" Oct 30 21:28:01.437281 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:01.437244 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdwxn\" (UniqueName: \"kubernetes.io/projected/7f05e57c-842d-4aac-b454-10ca04014727-kube-api-access-xdwxn\") pod \"payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj\" (UID: \"7f05e57c-842d-4aac-b454-10ca04014727\") " pod="openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj" Oct 30 21:28:01.437638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:01.437612 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23944deb-c056-466d-b79d-d56a257d98df-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk\" (UID: \"23944deb-c056-466d-b79d-d56a257d98df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk" Oct 30 21:28:01.437823 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:01.437806 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f05e57c-842d-4aac-b454-10ca04014727-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj\" (UID: \"7f05e57c-842d-4aac-b454-10ca04014727\") " pod="openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj" Oct 30 21:28:01.437993 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:01.437972 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45shf\" (UniqueName: \"kubernetes.io/projected/23944deb-c056-466d-b79d-d56a257d98df-kube-api-access-45shf\") pod \"payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk\" (UID: \"23944deb-c056-466d-b79d-d56a257d98df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk" Oct 30 21:28:01.492911 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:01.492054 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj"] Oct 30 21:28:01.586056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:01.575320 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdwxn\" (UniqueName: \"kubernetes.io/projected/7f05e57c-842d-4aac-b454-10ca04014727-kube-api-access-xdwxn\") pod \"payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj\" (UID: \"7f05e57c-842d-4aac-b454-10ca04014727\") " pod="openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj" Oct 30 21:28:01.586056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:01.575495 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23944deb-c056-466d-b79d-d56a257d98df-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk\" (UID: \"23944deb-c056-466d-b79d-d56a257d98df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk" Oct 30 21:28:01.586056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:01.575555 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f05e57c-842d-4aac-b454-10ca04014727-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj\" (UID: \"7f05e57c-842d-4aac-b454-10ca04014727\") " pod="openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj" Oct 30 21:28:01.586056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:01.575618 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45shf\" (UniqueName: \"kubernetes.io/projected/23944deb-c056-466d-b79d-d56a257d98df-kube-api-access-45shf\") pod \"payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk\" (UID: \"23944deb-c056-466d-b79d-d56a257d98df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk" Oct 30 21:28:01.586056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:01.577351 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23944deb-c056-466d-b79d-d56a257d98df-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk\" (UID: \"23944deb-c056-466d-b79d-d56a257d98df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk" Oct 30 21:28:01.625005 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:01.624323 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45shf\" (UniqueName: \"kubernetes.io/projected/23944deb-c056-466d-b79d-d56a257d98df-kube-api-access-45shf\") pod \"payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk\" (UID: \"23944deb-c056-466d-b79d-d56a257d98df\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk" Oct 30 21:28:01.644055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:01.642383 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f05e57c-842d-4aac-b454-10ca04014727-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj\" (UID: \"7f05e57c-842d-4aac-b454-10ca04014727\") " pod="openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj" Oct 30 21:28:01.689091 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:01.688382 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk" Oct 30 21:28:01.781456 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:01.781419 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdwxn\" (UniqueName: \"kubernetes.io/projected/7f05e57c-842d-4aac-b454-10ca04014727-kube-api-access-xdwxn\") pod \"payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj\" (UID: \"7f05e57c-842d-4aac-b454-10ca04014727\") " pod="openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj" Oct 30 21:28:02.031204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.031163 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj" Oct 30 21:28:02.166078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.162122 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk"] Oct 30 21:28:02.362780 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.362748 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj"] Oct 30 21:28:02.363610 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:28:02.363583 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f05e57c_842d_4aac_b454_10ca04014727.slice/crio-e9800f40afb497d7e534d77f244eccd9eb82cba46cbab5b3745ad5f47729b913 WatchSource:0}: Error finding container e9800f40afb497d7e534d77f244eccd9eb82cba46cbab5b3745ad5f47729b913: Status 404 returned error can't find the container with id e9800f40afb497d7e534d77f244eccd9eb82cba46cbab5b3745ad5f47729b913 Oct 30 21:28:02.417206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.417129 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w"] Oct 30 21:28:02.422933 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.422909 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" Oct 30 21:28:02.449046 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.449013 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-3b28bc-serving-cert\"" Oct 30 21:28:02.450497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.450471 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w"] Oct 30 21:28:02.542455 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.542420 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87506f32-cd27-497b-be39-ef8ae28faef0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w\" (UID: \"87506f32-cd27-497b-be39-ef8ae28faef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" Oct 30 21:28:02.542647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.542552 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l69h\" (UniqueName: \"kubernetes.io/projected/87506f32-cd27-497b-be39-ef8ae28faef0-kube-api-access-9l69h\") pod \"maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w\" (UID: \"87506f32-cd27-497b-be39-ef8ae28faef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" Oct 30 21:28:02.542726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.542704 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87506f32-cd27-497b-be39-ef8ae28faef0-maas-api-tls\") pod \"maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w\" (UID: \"87506f32-cd27-497b-be39-ef8ae28faef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" Oct 30 21:28:02.542780 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.542752 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87506f32-cd27-497b-be39-ef8ae28faef0-metrics-certs\") pod \"maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w\" (UID: \"87506f32-cd27-497b-be39-ef8ae28faef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" Oct 30 21:28:02.645571 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.644177 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87506f32-cd27-497b-be39-ef8ae28faef0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w\" (UID: \"87506f32-cd27-497b-be39-ef8ae28faef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" Oct 30 21:28:02.645571 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.644295 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l69h\" (UniqueName: \"kubernetes.io/projected/87506f32-cd27-497b-be39-ef8ae28faef0-kube-api-access-9l69h\") pod \"maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w\" (UID: \"87506f32-cd27-497b-be39-ef8ae28faef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" Oct 30 21:28:02.645571 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.644447 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87506f32-cd27-497b-be39-ef8ae28faef0-maas-api-tls\") pod \"maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w\" (UID: \"87506f32-cd27-497b-be39-ef8ae28faef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" Oct 30 21:28:02.645571 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.644500 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87506f32-cd27-497b-be39-ef8ae28faef0-metrics-certs\") pod \"maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w\" (UID: \"87506f32-cd27-497b-be39-ef8ae28faef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" Oct 30 21:28:02.647040 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.646997 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87506f32-cd27-497b-be39-ef8ae28faef0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w\" (UID: \"87506f32-cd27-497b-be39-ef8ae28faef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" Oct 30 21:28:02.650263 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.650227 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87506f32-cd27-497b-be39-ef8ae28faef0-maas-api-tls\") pod \"maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w\" (UID: \"87506f32-cd27-497b-be39-ef8ae28faef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" Oct 30 21:28:02.650263 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.650256 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87506f32-cd27-497b-be39-ef8ae28faef0-metrics-certs\") pod \"maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w\" (UID: \"87506f32-cd27-497b-be39-ef8ae28faef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" Oct 30 21:28:02.677312 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.677210 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l69h\" (UniqueName: \"kubernetes.io/projected/87506f32-cd27-497b-be39-ef8ae28faef0-kube-api-access-9l69h\") pod \"maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w\" (UID: \"87506f32-cd27-497b-be39-ef8ae28faef0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" Oct 30 21:28:02.749673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.749631 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" Oct 30 21:28:02.802773 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.801462 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj" event={"ID":"7f05e57c-842d-4aac-b454-10ca04014727","Type":"ContainerStarted","Data":"a19cb5d2133186dd977067429408f91404bb6231735c6dc3b243ec0cf3fa4b93"} Oct 30 21:28:02.802773 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.801536 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj" event={"ID":"7f05e57c-842d-4aac-b454-10ca04014727","Type":"ContainerStarted","Data":"e9800f40afb497d7e534d77f244eccd9eb82cba46cbab5b3745ad5f47729b913"} Oct 30 21:28:02.802773 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.802144 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj" Oct 30 21:28:02.804101 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.803668 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk" event={"ID":"23944deb-c056-466d-b79d-d56a257d98df","Type":"ContainerStarted","Data":"2e2c63096b7b3ea96867edb30086c08735468c0ff58091580e06ac1f49b57bee"} Oct 30 21:28:02.804101 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.803707 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk" event={"ID":"23944deb-c056-466d-b79d-d56a257d98df","Type":"ContainerStarted","Data":"0bddcf24f1e2fb1dbf7cd68539cb1fa7728a27b4241a6a02939bd63ee7699c64"} Oct 30 21:28:02.804101 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.803792 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk" Oct 30 21:28:02.909747 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.909684 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj" podStartSLOduration=1.909668288 podStartE2EDuration="1.909668288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:02.905349462 +0000 UTC m=+1670.026423209" watchObservedRunningTime="2026-10-30 21:28:02.909668288 +0000 UTC m=+1670.030742037" Oct 30 21:28:02.987196 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:02.987081 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk" podStartSLOduration=1.987065833 podStartE2EDuration="1.987065833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:02.985843856 +0000 UTC m=+1670.106917605" watchObservedRunningTime="2026-10-30 21:28:02.987065833 +0000 UTC m=+1670.108139579" Oct 30 21:28:03.129830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:03.129795 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w"] Oct 30 21:28:03.819375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:03.819330 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" event={"ID":"87506f32-cd27-497b-be39-ef8ae28faef0","Type":"ContainerStarted","Data":"80d2a56210390c907286427da1ea74cd2c4dcf8b16af5bb3efac2780dbe57d45"} Oct 30 21:28:03.819889 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:03.819400 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" event={"ID":"87506f32-cd27-497b-be39-ef8ae28faef0","Type":"ContainerStarted","Data":"df65641218f1b1f4ad01409d589f2854d770278f240c811e2fec87ff98c242a1"} Oct 30 21:28:03.820287 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:03.820258 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" Oct 30 21:28:03.917808 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:03.917748 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" podStartSLOduration=1.452524948 podStartE2EDuration="1.917728871s" totalImagesPullingTime="465.203923ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:03.907327589 +0000 UTC m=+1671.028401337" watchObservedRunningTime="2026-10-30 21:28:03.917728871 +0000 UTC m=+1671.038802625" Oct 30 21:28:05.838479 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:05.838391 4926 generic.go:356] "Generic (PLEG): container finished" podID="2eea6ad7-9d9b-4205-89d2-af24991331de" containerID="d43d73aee50bbddfb466a095c47aab5d3f4afef733d0c6c96e6b2c39dc15637d" exitCode=0 Oct 30 21:28:05.838479 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:05.838462 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" event={"ID":"2eea6ad7-9d9b-4205-89d2-af24991331de","Type":"ContainerDied","Data":"d43d73aee50bbddfb466a095c47aab5d3f4afef733d0c6c96e6b2c39dc15637d"} Oct 30 21:28:05.839135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:05.838962 4926 scope.go:122] "RemoveContainer" containerID="d43d73aee50bbddfb466a095c47aab5d3f4afef733d0c6c96e6b2c39dc15637d" Oct 30 21:28:05.840783 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:05.840751 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:28:05.840866 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:05.840824 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:28:05.840909 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:05.840870 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:28:06.849094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:06.849059 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" event={"ID":"2eea6ad7-9d9b-4205-89d2-af24991331de","Type":"ContainerStarted","Data":"7191f18d5a586ae57a6f22b17224e3217055ff9cffd5606b933ca51a4ca0d391"} Oct 30 21:28:07.524264 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:28:07.524225 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:28:07.760560 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:07.760529 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:28:07.760839 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:07.760815 4926 patch_prober.go:29] interesting pod/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:28:07.760892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:07.760864 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" podUID="2eea6ad7-9d9b-4205-89d2-af24991331de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 21:28:08.450670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:08.450634 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7" Oct 30 21:28:08.456697 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:08.456659 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk" Oct 30 21:28:08.760670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:08.760587 4926 patch_prober.go:29] interesting pod/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:28:08.760914 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:08.760888 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" podUID="2eea6ad7-9d9b-4205-89d2-af24991331de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 21:28:09.761441 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:09.761404 4926 patch_prober.go:29] interesting pod/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:28:09.761869 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:09.761464 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" podUID="2eea6ad7-9d9b-4205-89d2-af24991331de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 21:28:10.760972 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:10.760939 4926 patch_prober.go:29] interesting pod/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:28:10.761165 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:10.760990 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" podUID="2eea6ad7-9d9b-4205-89d2-af24991331de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 21:28:10.845092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:10.845059 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" Oct 30 21:28:11.760692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:11.760636 4926 patch_prober.go:29] interesting pod/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 30 21:28:11.760936 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:11.760749 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" podUID="2eea6ad7-9d9b-4205-89d2-af24991331de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 30 21:28:12.766148 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:12.766118 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:28:12.766679 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:12.766635 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:28:12.767459 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:12.767432 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:28:13.822314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:13.822287 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj" Oct 30 21:28:13.822734 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:13.822370 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk" Oct 30 21:28:16.760881 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:16.760835 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5"] Oct 30 21:28:16.765504 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:16.765470 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:16.796865 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:16.796824 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-093daa-openshift-default-dockercfg-wbwdl\"" Oct 30 21:28:16.806800 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:16.805247 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5"] Oct 30 21:28:16.908106 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:16.908065 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-istio-data\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:16.908335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:16.908163 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2bp9\" (UniqueName: \"kubernetes.io/projected/7b2c155a-636e-4760-9585-e8923ea0318a-kube-api-access-x2bp9\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:16.908402 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:16.908381 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b2c155a-636e-4760-9585-e8923ea0318a-istio-podinfo\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:16.908560 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:16.908542 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-credential-socket\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:16.908623 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:16.908582 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-workload-socket\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:16.908708 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:16.908684 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-istio-envoy\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:16.908767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:16.908757 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-workload-certs\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:16.908907 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:16.908884 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b2c155a-636e-4760-9585-e8923ea0318a-istio-token\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:16.908962 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:16.908947 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b2c155a-636e-4760-9585-e8923ea0318a-istiod-ca-cert\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:17.010274 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.010238 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b2c155a-636e-4760-9585-e8923ea0318a-istio-token\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:17.010545 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.010291 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b2c155a-636e-4760-9585-e8923ea0318a-istiod-ca-cert\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:17.010545 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.010317 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-istio-data\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:17.010545 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.010336 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2bp9\" (UniqueName: \"kubernetes.io/projected/7b2c155a-636e-4760-9585-e8923ea0318a-kube-api-access-x2bp9\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:17.010545 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.010494 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b2c155a-636e-4760-9585-e8923ea0318a-istio-podinfo\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:17.010989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.010919 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-credential-socket\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:17.010989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.010972 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-workload-socket\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:17.011194 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.011017 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-istio-envoy\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:17.011194 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.011162 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-workload-certs\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:17.011407 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.011338 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-credential-socket\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:17.011475 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.011449 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-workload-socket\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:17.011554 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.011523 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-workload-certs\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:17.011685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.011584 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-istio-data\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:17.012162 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.012136 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b2c155a-636e-4760-9585-e8923ea0318a-istiod-ca-cert\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:17.013757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.013733 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-istio-envoy\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:17.014118 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.014094 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b2c155a-636e-4760-9585-e8923ea0318a-istio-podinfo\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:17.047422 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.047387 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b2c155a-636e-4760-9585-e8923ea0318a-istio-token\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:17.058689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.058656 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2bp9\" (UniqueName: \"kubernetes.io/projected/7b2c155a-636e-4760-9585-e8923ea0318a-kube-api-access-x2bp9\") pod \"e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:17.084094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.084054 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:17.474114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.474079 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5"] Oct 30 21:28:17.474521 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:28:17.474496 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2c155a_636e_4760_9585_e8923ea0318a.slice/crio-fd3e1de6cd321ee6035947851bc4fa1be82580ccfce6372bd0b715beceea514d WatchSource:0}: Error finding container fd3e1de6cd321ee6035947851bc4fa1be82580ccfce6372bd0b715beceea514d: Status 404 returned error can't find the container with id fd3e1de6cd321ee6035947851bc4fa1be82580ccfce6372bd0b715beceea514d Oct 30 21:28:17.478602 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.478563 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:28:17.478696 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.478652 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:28:17.478739 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.478693 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:28:17.994541 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.994120 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" event={"ID":"7b2c155a-636e-4760-9585-e8923ea0318a","Type":"ContainerStarted","Data":"7ecbe8147b41d84af35b1c51a75fcb688faab6af494dbe446bc8bee4595abf01"} Oct 30 21:28:17.994541 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:17.994233 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" event={"ID":"7b2c155a-636e-4760-9585-e8923ea0318a","Type":"ContainerStarted","Data":"fd3e1de6cd321ee6035947851bc4fa1be82580ccfce6372bd0b715beceea514d"} Oct 30 21:28:18.050360 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:18.050293 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" podStartSLOduration=2.050270834 podStartE2EDuration="2.050270834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:18.043434347 +0000 UTC m=+1685.164508093" watchObservedRunningTime="2026-10-30 21:28:18.050270834 +0000 UTC m=+1685.171344583" Oct 30 21:28:18.085217 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:18.085178 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:18.088276 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:18.088238 4926 patch_prober.go:29] interesting pod/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 21:28:18.088459 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:18.088299 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" podUID="7b2c155a-636e-4760-9585-e8923ea0318a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 21:28:19.088058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:19.086362 4926 patch_prober.go:29] interesting pod/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 21:28:19.088058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:19.086429 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" podUID="7b2c155a-636e-4760-9585-e8923ea0318a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 21:28:20.089055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:20.088417 4926 patch_prober.go:29] interesting pod/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 21:28:20.089055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:20.088483 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" podUID="7b2c155a-636e-4760-9585-e8923ea0318a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 21:28:21.090056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:21.089400 4926 patch_prober.go:29] interesting pod/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 21:28:21.090056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:21.089504 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" podUID="7b2c155a-636e-4760-9585-e8923ea0318a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 21:28:22.088057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.086666 4926 patch_prober.go:29] interesting pod/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 21:28:22.088057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.086728 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" podUID="7b2c155a-636e-4760-9585-e8923ea0318a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 21:28:22.437737 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.437706 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr"] Oct 30 21:28:22.441869 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.441849 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:22.517054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.511373 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1eab65\"/\"test-model-1eab65-kserve-self-signed-certs\"" Oct 30 21:28:22.517054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.511657 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1eab65\"/\"kube-root-ca.crt\"" Oct 30 21:28:22.517054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.511746 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1eab65\"/\"default-dockercfg-92pnk\"" Oct 30 21:28:22.517054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.513610 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1eab65\"/\"openshift-service-ca.crt\"" Oct 30 21:28:22.527084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.523175 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr"] Oct 30 21:28:22.538277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.532100 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-model-cache\") pod \"test-model-1eab65-kserve-75f9fc9ccf-vxrtr\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:22.538277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.532143 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r52cq\" (UniqueName: \"kubernetes.io/projected/6df032f2-153b-48f9-9e31-3ad79d10bcfc-kube-api-access-r52cq\") pod \"test-model-1eab65-kserve-75f9fc9ccf-vxrtr\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:22.538277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.532194 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-dshm\") pod \"test-model-1eab65-kserve-75f9fc9ccf-vxrtr\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:22.538277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.532228 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-home\") pod \"test-model-1eab65-kserve-75f9fc9ccf-vxrtr\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:22.538277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.532490 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-tmp-dir\") pod \"test-model-1eab65-kserve-75f9fc9ccf-vxrtr\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:22.538277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.532528 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6df032f2-153b-48f9-9e31-3ad79d10bcfc-tls-certs\") pod \"test-model-1eab65-kserve-75f9fc9ccf-vxrtr\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:22.634312 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.634252 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-tmp-dir\") pod \"test-model-1eab65-kserve-75f9fc9ccf-vxrtr\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:22.634520 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.634339 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6df032f2-153b-48f9-9e31-3ad79d10bcfc-tls-certs\") pod \"test-model-1eab65-kserve-75f9fc9ccf-vxrtr\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:22.634520 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.634464 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-model-cache\") pod \"test-model-1eab65-kserve-75f9fc9ccf-vxrtr\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:22.634520 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.634489 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r52cq\" (UniqueName: \"kubernetes.io/projected/6df032f2-153b-48f9-9e31-3ad79d10bcfc-kube-api-access-r52cq\") pod \"test-model-1eab65-kserve-75f9fc9ccf-vxrtr\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:22.634703 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.634543 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-dshm\") pod \"test-model-1eab65-kserve-75f9fc9ccf-vxrtr\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:22.634703 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.634578 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-home\") pod \"test-model-1eab65-kserve-75f9fc9ccf-vxrtr\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:22.635270 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.635238 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-model-cache\") pod \"test-model-1eab65-kserve-75f9fc9ccf-vxrtr\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:22.635707 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.635655 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-tmp-dir\") pod \"test-model-1eab65-kserve-75f9fc9ccf-vxrtr\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:22.636157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.636130 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-home\") pod \"test-model-1eab65-kserve-75f9fc9ccf-vxrtr\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:22.637412 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.637386 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-dshm\") pod \"test-model-1eab65-kserve-75f9fc9ccf-vxrtr\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:22.637717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.637697 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6df032f2-153b-48f9-9e31-3ad79d10bcfc-tls-certs\") pod \"test-model-1eab65-kserve-75f9fc9ccf-vxrtr\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:22.731490 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.731344 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r52cq\" (UniqueName: \"kubernetes.io/projected/6df032f2-153b-48f9-9e31-3ad79d10bcfc-kube-api-access-r52cq\") pod \"test-model-1eab65-kserve-75f9fc9ccf-vxrtr\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:22.757054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:22.756778 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:23.091271 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:23.090180 4926 patch_prober.go:29] interesting pod/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 21:28:23.091271 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:23.090247 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" podUID="7b2c155a-636e-4760-9585-e8923ea0318a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 21:28:23.359549 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:23.359516 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr"] Oct 30 21:28:23.361295 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:28:23.361251 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6df032f2_153b_48f9_9e31_3ad79d10bcfc.slice/crio-2fd98c033e69b7c292174101eebfa1eaf9a5d5b6b340e9a31a763e4a92c4b247 WatchSource:0}: Error finding container 2fd98c033e69b7c292174101eebfa1eaf9a5d5b6b340e9a31a763e4a92c4b247: Status 404 returned error can't find the container with id 2fd98c033e69b7c292174101eebfa1eaf9a5d5b6b340e9a31a763e4a92c4b247 Oct 30 21:28:24.068152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:24.068105 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" event={"ID":"6df032f2-153b-48f9-9e31-3ad79d10bcfc","Type":"ContainerStarted","Data":"7d79e9f657cb7324eb7086d2e2264738aa0b250dd5c3bf7af7cd8fd29090bd9e"} Oct 30 21:28:24.068735 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:24.068167 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" event={"ID":"6df032f2-153b-48f9-9e31-3ad79d10bcfc","Type":"ContainerStarted","Data":"2fd98c033e69b7c292174101eebfa1eaf9a5d5b6b340e9a31a763e4a92c4b247"} Oct 30 21:28:24.084713 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:24.084664 4926 patch_prober.go:29] interesting pod/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" start-of-body= Oct 30 21:28:24.084933 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:24.084736 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" podUID="7b2c155a-636e-4760-9585-e8923ea0318a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.154:15021/healthz/ready\": dial tcp 10.132.0.154:15021: connect: connection refused" Oct 30 21:28:24.129540 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:24.129477 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" podStartSLOduration=2.129459925 podStartE2EDuration="2.129459925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:24.12681823 +0000 UTC m=+1691.247891975" watchObservedRunningTime="2026-10-30 21:28:24.129459925 +0000 UTC m=+1691.250533692" Oct 30 21:28:24.757636 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:24.757595 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:25.090796 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:25.090710 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:25.091303 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:25.091275 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:25.092102 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:25.092081 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:28:25.468061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:25.458815 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:28:31.687435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:31.687396 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf"] Oct 30 21:28:31.702295 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:31.702258 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-517366-x4mlf"] Oct 30 21:28:32.757703 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:32.757667 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:32.771146 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:32.771110 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:33.033273 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:33.033157 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b30f1ce4-1915-46fa-8393-9e3ffbf922ea" path="/var/lib/kubelet/pods/b30f1ce4-1915-46fa-8393-9e3ffbf922ea/volumes" Oct 30 21:28:33.168439 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:33.168400 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:28:37.492727 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:28:37.492676 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:28:40.532517 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.532476 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9"] Oct 30 21:28:40.540356 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.540327 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.572194 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.572119 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-abf04b-openshift-default-dockercfg-knqgx\"" Oct 30 21:28:40.579062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.575551 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-envoy\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.579062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.575609 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-workload-socket\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.579062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.575636 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8l2z\" (UniqueName: \"kubernetes.io/projected/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-kube-api-access-h8l2z\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.579062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.575739 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istiod-ca-cert\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.579062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.575798 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-credential-socket\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.579062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.575893 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-podinfo\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.579062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.575983 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-workload-certs\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.579062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.576021 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-token\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.579062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.576073 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-data\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.579062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.576813 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9"] Oct 30 21:28:40.678019 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.677981 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-workload-certs\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.678276 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.678259 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-token\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.678364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.678353 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-data\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.678503 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.678493 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-envoy\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.678579 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.678570 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-workload-socket\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.678663 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.678650 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8l2z\" (UniqueName: \"kubernetes.io/projected/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-kube-api-access-h8l2z\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.678885 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.678866 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istiod-ca-cert\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.679008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.678986 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-data\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.679103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.678704 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-workload-certs\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.679311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.679278 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-workload-socket\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.679598 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.679572 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-credential-socket\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.679718 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.679700 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-podinfo\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.679858 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.679830 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istiod-ca-cert\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.680130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.680112 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-credential-socket\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.682235 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.682211 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-envoy\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.685836 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.685801 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-podinfo\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.709849 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.704805 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-token\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.709849 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.709843 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8l2z\" (UniqueName: \"kubernetes.io/projected/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-kube-api-access-h8l2z\") pod \"e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:40.899677 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:40.899578 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:41.252872 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:28:41.252830 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12f6bdd5_2b23_4b38_a5d1_2b5fc498c9c4.slice/crio-fb5e8e4bd3a26b1852c909ecb395b9fef4c7dc5bb94e5156b05a2e504fa1f4de WatchSource:0}: Error finding container fb5e8e4bd3a26b1852c909ecb395b9fef4c7dc5bb94e5156b05a2e504fa1f4de: Status 404 returned error can't find the container with id fb5e8e4bd3a26b1852c909ecb395b9fef4c7dc5bb94e5156b05a2e504fa1f4de Oct 30 21:28:41.253511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:41.253485 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9"] Oct 30 21:28:41.257076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:41.256506 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:28:41.257076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:41.256594 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:28:41.257076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:41.256643 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:28:41.295256 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:41.295227 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" event={"ID":"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4","Type":"ContainerStarted","Data":"fb5e8e4bd3a26b1852c909ecb395b9fef4c7dc5bb94e5156b05a2e504fa1f4de"} Oct 30 21:28:42.309420 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:42.309375 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" event={"ID":"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4","Type":"ContainerStarted","Data":"0a21174f122abf4b93eef30871d437bc2a4f7bf191945c9aaa103461c74a43da"} Oct 30 21:28:42.392481 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:42.392405 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" podStartSLOduration=2.392383084 podStartE2EDuration="2.392383084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:28:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:28:42.377183637 +0000 UTC m=+1709.498257384" watchObservedRunningTime="2026-10-30 21:28:42.392383084 +0000 UTC m=+1709.513456830" Oct 30 21:28:42.900245 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:42.900206 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:42.908771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:42.908737 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:43.319555 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:43.319525 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:28:43.321536 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:28:43.321500 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:29:01.937177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:01.937139 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b"] Oct 30 21:29:01.945080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:01.945054 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" Oct 30 21:29:01.963691 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:01.963668 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-200d11-serving-cert\"" Oct 30 21:29:01.969443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:01.969408 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b"] Oct 30 21:29:02.071648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:02.071607 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-057c1dfd-pqhtt"] Oct 30 21:29:02.078887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:02.078851 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-057c1dfd-pqhtt"] Oct 30 21:29:02.082465 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:02.082436 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be8482db-1f53-4f48-bf7a-fd17a081c6f9-metrics-certs\") pod \"maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b\" (UID: \"be8482db-1f53-4f48-bf7a-fd17a081c6f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" Oct 30 21:29:02.082714 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:02.082690 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9cvd\" (UniqueName: \"kubernetes.io/projected/be8482db-1f53-4f48-bf7a-fd17a081c6f9-kube-api-access-k9cvd\") pod \"maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b\" (UID: \"be8482db-1f53-4f48-bf7a-fd17a081c6f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" Oct 30 21:29:02.082890 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:02.082862 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be8482db-1f53-4f48-bf7a-fd17a081c6f9-maas-api-tls\") pod \"maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b\" (UID: \"be8482db-1f53-4f48-bf7a-fd17a081c6f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" Oct 30 21:29:02.083028 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:02.083011 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8482db-1f53-4f48-bf7a-fd17a081c6f9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b\" (UID: \"be8482db-1f53-4f48-bf7a-fd17a081c6f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" Oct 30 21:29:02.187062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:02.185001 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8482db-1f53-4f48-bf7a-fd17a081c6f9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b\" (UID: \"be8482db-1f53-4f48-bf7a-fd17a081c6f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" Oct 30 21:29:02.187062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:02.185132 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be8482db-1f53-4f48-bf7a-fd17a081c6f9-metrics-certs\") pod \"maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b\" (UID: \"be8482db-1f53-4f48-bf7a-fd17a081c6f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" Oct 30 21:29:02.187062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:02.185322 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9cvd\" (UniqueName: \"kubernetes.io/projected/be8482db-1f53-4f48-bf7a-fd17a081c6f9-kube-api-access-k9cvd\") pod \"maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b\" (UID: \"be8482db-1f53-4f48-bf7a-fd17a081c6f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" Oct 30 21:29:02.187062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:02.185424 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be8482db-1f53-4f48-bf7a-fd17a081c6f9-maas-api-tls\") pod \"maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b\" (UID: \"be8482db-1f53-4f48-bf7a-fd17a081c6f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" Oct 30 21:29:02.211128 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:02.200470 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be8482db-1f53-4f48-bf7a-fd17a081c6f9-maas-api-tls\") pod \"maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b\" (UID: \"be8482db-1f53-4f48-bf7a-fd17a081c6f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" Oct 30 21:29:02.211128 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:02.200528 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8482db-1f53-4f48-bf7a-fd17a081c6f9-openshift-service-ca\") pod \"maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b\" (UID: \"be8482db-1f53-4f48-bf7a-fd17a081c6f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" Oct 30 21:29:02.211128 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:02.205637 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be8482db-1f53-4f48-bf7a-fd17a081c6f9-metrics-certs\") pod \"maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b\" (UID: \"be8482db-1f53-4f48-bf7a-fd17a081c6f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" Oct 30 21:29:02.232582 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:02.232547 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9cvd\" (UniqueName: \"kubernetes.io/projected/be8482db-1f53-4f48-bf7a-fd17a081c6f9-kube-api-access-k9cvd\") pod \"maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b\" (UID: \"be8482db-1f53-4f48-bf7a-fd17a081c6f9\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" Oct 30 21:29:02.267073 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:02.267005 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" Oct 30 21:29:02.609343 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:29:02.609306 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8482db_1f53_4f48_bf7a_fd17a081c6f9.slice/crio-a75a4d41f7a19cc2dec6be6d27b41dc889f849ed7f2c64de66e88e2672e6753e WatchSource:0}: Error finding container a75a4d41f7a19cc2dec6be6d27b41dc889f849ed7f2c64de66e88e2672e6753e: Status 404 returned error can't find the container with id a75a4d41f7a19cc2dec6be6d27b41dc889f849ed7f2c64de66e88e2672e6753e Oct 30 21:29:02.610545 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:02.610513 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b"] Oct 30 21:29:03.035625 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:03.035592 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba0927fd-aba0-4ab2-8e77-85c03e39dcbd" path="/var/lib/kubelet/pods/ba0927fd-aba0-4ab2-8e77-85c03e39dcbd/volumes" Oct 30 21:29:03.565426 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:03.564209 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" event={"ID":"be8482db-1f53-4f48-bf7a-fd17a081c6f9","Type":"ContainerStarted","Data":"c6e690d2ef345bc09cec63bd149db7b5ee5ce7a85af07b25df3aa05183b6b653"} Oct 30 21:29:03.565426 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:03.564283 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" event={"ID":"be8482db-1f53-4f48-bf7a-fd17a081c6f9","Type":"ContainerStarted","Data":"a75a4d41f7a19cc2dec6be6d27b41dc889f849ed7f2c64de66e88e2672e6753e"} Oct 30 21:29:03.565712 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:03.565442 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" Oct 30 21:29:03.632309 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:03.632223 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" podStartSLOduration=2.333820031 podStartE2EDuration="2.632202123s" totalImagesPullingTime="298.382092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:03.621014908 +0000 UTC m=+1730.742088659" watchObservedRunningTime="2026-10-30 21:29:03.632202123 +0000 UTC m=+1730.753275874" Oct 30 21:29:03.797311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:03.797274 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg"] Oct 30 21:29:03.797532 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:03.797511 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" podUID="b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70" containerName="istio-proxy" containerID="cri-o://8ac910cc08ef3b7b28ecadebbbd09f8927f9505427669f6e06028fb949ea4069" gracePeriod=30 Oct 30 21:29:04.009800 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.009757 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp"] Oct 30 21:29:04.019965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.019932 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp"] Oct 30 21:29:04.037058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.028222 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:04.055313 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.055286 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-290ede\"/\"openshift-service-ca.crt\"" Oct 30 21:29:04.055560 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.055286 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-290ede\"/\"kube-root-ca.crt\"" Oct 30 21:29:04.059430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.059384 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-290ede\"/\"default-dockercfg-jh6ds\"" Oct 30 21:29:04.059700 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.059413 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-290ede\"/\"test-model-290ede-kserve-self-signed-certs\"" Oct 30 21:29:04.129439 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.129404 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-dshm\") pod \"test-model-290ede-kserve-dbdd9f749-jvjdp\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:04.129668 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.129584 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-model-cache\") pod \"test-model-290ede-kserve-dbdd9f749-jvjdp\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:04.130117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.130083 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt7zm\" (UniqueName: \"kubernetes.io/projected/f05d1479-4fa1-4640-a6f5-59f43084b401-kube-api-access-kt7zm\") pod \"test-model-290ede-kserve-dbdd9f749-jvjdp\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:04.130323 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.130297 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-tmp-dir\") pod \"test-model-290ede-kserve-dbdd9f749-jvjdp\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:04.130396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.130337 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-home\") pod \"test-model-290ede-kserve-dbdd9f749-jvjdp\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:04.130396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.130364 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f05d1479-4fa1-4640-a6f5-59f43084b401-tls-certs\") pod \"test-model-290ede-kserve-dbdd9f749-jvjdp\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:04.232368 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.232322 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt7zm\" (UniqueName: \"kubernetes.io/projected/f05d1479-4fa1-4640-a6f5-59f43084b401-kube-api-access-kt7zm\") pod \"test-model-290ede-kserve-dbdd9f749-jvjdp\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:04.232579 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.232392 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-tmp-dir\") pod \"test-model-290ede-kserve-dbdd9f749-jvjdp\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:04.232579 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.232417 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-home\") pod \"test-model-290ede-kserve-dbdd9f749-jvjdp\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:04.232579 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.232441 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f05d1479-4fa1-4640-a6f5-59f43084b401-tls-certs\") pod \"test-model-290ede-kserve-dbdd9f749-jvjdp\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:04.232770 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.232587 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-dshm\") pod \"test-model-290ede-kserve-dbdd9f749-jvjdp\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:04.232829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.232779 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-model-cache\") pod \"test-model-290ede-kserve-dbdd9f749-jvjdp\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:04.232889 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.232870 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-tmp-dir\") pod \"test-model-290ede-kserve-dbdd9f749-jvjdp\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:04.232952 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.232919 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-home\") pod \"test-model-290ede-kserve-dbdd9f749-jvjdp\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:04.233152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.233125 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-model-cache\") pod \"test-model-290ede-kserve-dbdd9f749-jvjdp\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:04.235347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.235323 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f05d1479-4fa1-4640-a6f5-59f43084b401-tls-certs\") pod \"test-model-290ede-kserve-dbdd9f749-jvjdp\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:04.235347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.235344 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-dshm\") pod \"test-model-290ede-kserve-dbdd9f749-jvjdp\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:04.269921 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.269834 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt7zm\" (UniqueName: \"kubernetes.io/projected/f05d1479-4fa1-4640-a6f5-59f43084b401-kube-api-access-kt7zm\") pod \"test-model-290ede-kserve-dbdd9f749-jvjdp\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:04.365537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.365498 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:04.953723 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:29:04.953694 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf05d1479_4fa1_4640_a6f5_59f43084b401.slice/crio-c04a20ff1da9618a0d8e259b5e082129a8681a67d2377f283494fae68d50576f WatchSource:0}: Error finding container c04a20ff1da9618a0d8e259b5e082129a8681a67d2377f283494fae68d50576f: Status 404 returned error can't find the container with id c04a20ff1da9618a0d8e259b5e082129a8681a67d2377f283494fae68d50576f Oct 30 21:29:04.953834 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:04.953753 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp"] Oct 30 21:29:05.730544 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:05.730455 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" event={"ID":"f05d1479-4fa1-4640-a6f5-59f43084b401","Type":"ContainerStarted","Data":"1d1cd4004e14bb6aac1c584def6c23b4ac39f1a6b37c27e7db0bc8fd137b4ccc"} Oct 30 21:29:05.730544 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:05.730509 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" event={"ID":"f05d1479-4fa1-4640-a6f5-59f43084b401","Type":"ContainerStarted","Data":"c04a20ff1da9618a0d8e259b5e082129a8681a67d2377f283494fae68d50576f"} Oct 30 21:29:05.780688 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:05.780620 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" podStartSLOduration=2.780603028 podStartE2EDuration="2.780603028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:05.777549261 +0000 UTC m=+1732.898623004" watchObservedRunningTime="2026-10-30 21:29:05.780603028 +0000 UTC m=+1732.901676775" Oct 30 21:29:06.366398 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:06.366355 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:07.488767 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:29:07.488689 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:29:08.968224 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:08.966464 4926 patch_prober.go:29] interesting pod/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 30 21:29:08.968224 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:08.966516 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" podUID="b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 30 21:29:09.131019 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.130989 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:29:09.261787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.261749 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istiod-ca-cert\") pod \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " Oct 30 21:29:09.261976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.261884 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-kube-api-access-w6mkm\" (UniqueName: \"kubernetes.io/projected/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-kube-api-access-w6mkm\") pod \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " Oct 30 21:29:09.261976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.261939 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-credential-socket\") pod \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " Oct 30 21:29:09.262127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.261980 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-envoy\") pod \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " Oct 30 21:29:09.262189 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.262121 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-workload-certs\") pod \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " Oct 30 21:29:09.262189 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.262164 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-podinfo\") pod \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " Oct 30 21:29:09.262189 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.262175 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istiod-ca-cert" pod "b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70" (UID: "b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:29:09.262358 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.262302 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-data\") pod \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " Oct 30 21:29:09.262416 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.262355 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-workload-socket\") pod \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " Oct 30 21:29:09.262481 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.262433 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-token\" (UniqueName: \"kubernetes.io/projected/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-token\") pod \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\" (UID: \"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70\") " Oct 30 21:29:09.262747 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.262713 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-workload-socket" pod "b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70" (UID: "b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:29:09.262871 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.262838 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-data" pod "b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70" (UID: "b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:29:09.262937 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.262908 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-workload-certs" pod "b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70" (UID: "b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:29:09.263575 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.263545 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:29:09.263683 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.263577 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:29:09.263683 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.263593 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:29:09.263683 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.263605 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:29:09.265999 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.265759 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-token" pod "b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70" (UID: "b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:09.265999 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.265879 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-podinfo" pod "b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70" (UID: "b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:29:09.272319 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.272292 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-credential-socket" pod "b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70" (UID: "b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:29:09.294090 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.291718 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-envoy" pod "b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70" (UID: "b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:29:09.300367 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.300328 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-kube-api-access-w6mkm" pod "b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70" (UID: "b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70"). InnerVolumeSpecName "kube-api-access-w6mkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:29:09.364964 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.364877 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6mkm\" (UniqueName: \"kubernetes.io/projected/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-kube-api-access-w6mkm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:29:09.364964 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.364909 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:29:09.364964 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.364918 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:29:09.364964 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.364927 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:29:09.364964 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.364935 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:29:09.770226 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.770191 4926 generic.go:356] "Generic (PLEG): container finished" podID="b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70" containerID="8ac910cc08ef3b7b28ecadebbbd09f8927f9505427669f6e06028fb949ea4069" exitCode=0 Oct 30 21:29:09.770412 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.770251 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" Oct 30 21:29:09.770412 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.770295 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" event={"ID":"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70","Type":"ContainerDied","Data":"8ac910cc08ef3b7b28ecadebbbd09f8927f9505427669f6e06028fb949ea4069"} Oct 30 21:29:09.770412 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.770356 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg" event={"ID":"b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70","Type":"ContainerDied","Data":"574e1565b274d545439a79d559532448a46a1e6fee52384eb45ac4eedf467cf2"} Oct 30 21:29:09.770412 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.770386 4926 scope.go:122] "RemoveContainer" containerID="8ac910cc08ef3b7b28ecadebbbd09f8927f9505427669f6e06028fb949ea4069" Oct 30 21:29:09.784449 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.784425 4926 scope.go:122] "RemoveContainer" containerID="8ac910cc08ef3b7b28ecadebbbd09f8927f9505427669f6e06028fb949ea4069" Oct 30 21:29:09.784804 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:29:09.784773 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ac910cc08ef3b7b28ecadebbbd09f8927f9505427669f6e06028fb949ea4069\": container with ID starting with 8ac910cc08ef3b7b28ecadebbbd09f8927f9505427669f6e06028fb949ea4069 not found: ID does not exist" containerID="8ac910cc08ef3b7b28ecadebbbd09f8927f9505427669f6e06028fb949ea4069" Oct 30 21:29:09.784908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.784815 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ac910cc08ef3b7b28ecadebbbd09f8927f9505427669f6e06028fb949ea4069"} err="failed to get container status \"8ac910cc08ef3b7b28ecadebbbd09f8927f9505427669f6e06028fb949ea4069\": rpc error: code = NotFound desc = could not find container \"8ac910cc08ef3b7b28ecadebbbd09f8927f9505427669f6e06028fb949ea4069\": container with ID starting with 8ac910cc08ef3b7b28ecadebbbd09f8927f9505427669f6e06028fb949ea4069 not found: ID does not exist" Oct 30 21:29:09.868013 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.867975 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg"] Oct 30 21:29:09.891425 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:09.891385 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-pcgrg"] Oct 30 21:29:10.743893 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:10.743862 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" Oct 30 21:29:11.022814 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:11.022732 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70" path="/var/lib/kubelet/pods/b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70/volumes" Oct 30 21:29:13.753192 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.753155 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22"] Oct 30 21:29:13.754104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.754080 4926 cpu_manager.go:414] "containerMap: removing container" podUID="b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70" containerName="istio-proxy" Oct 30 21:29:13.754104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.754103 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70" containerName="istio-proxy" Oct 30 21:29:13.754319 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.754304 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="b71625f0-cf7b-4c3e-8e7c-fe0c6d8d9d70" containerName="istio-proxy" Oct 30 21:29:13.759601 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.759579 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:13.779265 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.779225 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-79c24a\"/\"kube-root-ca.crt\"" Oct 30 21:29:13.779265 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.779240 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-79c24a\"/\"openshift-service-ca.crt\"" Oct 30 21:29:13.790648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.790611 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-79c24a\"/\"test-model-79c24a-kserve-self-signed-certs\"" Oct 30 21:29:13.790648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.790615 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-79c24a\"/\"default-dockercfg-4skgr\"" Oct 30 21:29:13.802091 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.798602 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22"] Oct 30 21:29:13.847995 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.847954 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-tmp-dir\") pod \"test-model-79c24a-kserve-7bf88d7d67-clv22\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:13.848259 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.848014 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khmjd\" (UniqueName: \"kubernetes.io/projected/50ced360-7983-4b5a-8b03-35714cd87a9c-kube-api-access-khmjd\") pod \"test-model-79c24a-kserve-7bf88d7d67-clv22\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:13.848259 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.848106 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50ced360-7983-4b5a-8b03-35714cd87a9c-tls-certs\") pod \"test-model-79c24a-kserve-7bf88d7d67-clv22\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:13.848259 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.848157 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-home\") pod \"test-model-79c24a-kserve-7bf88d7d67-clv22\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:13.848470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.848304 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-dshm\") pod \"test-model-79c24a-kserve-7bf88d7d67-clv22\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:13.848470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.848340 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-model-cache\") pod \"test-model-79c24a-kserve-7bf88d7d67-clv22\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:13.949991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.949943 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-tmp-dir\") pod \"test-model-79c24a-kserve-7bf88d7d67-clv22\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:13.950235 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.950014 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khmjd\" (UniqueName: \"kubernetes.io/projected/50ced360-7983-4b5a-8b03-35714cd87a9c-kube-api-access-khmjd\") pod \"test-model-79c24a-kserve-7bf88d7d67-clv22\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:13.950235 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.950079 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50ced360-7983-4b5a-8b03-35714cd87a9c-tls-certs\") pod \"test-model-79c24a-kserve-7bf88d7d67-clv22\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:13.950235 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.950127 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-home\") pod \"test-model-79c24a-kserve-7bf88d7d67-clv22\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:13.950429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.950267 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-dshm\") pod \"test-model-79c24a-kserve-7bf88d7d67-clv22\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:13.950429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.950301 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-model-cache\") pod \"test-model-79c24a-kserve-7bf88d7d67-clv22\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:13.950542 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.950483 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-tmp-dir\") pod \"test-model-79c24a-kserve-7bf88d7d67-clv22\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:13.950590 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.950548 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-home\") pod \"test-model-79c24a-kserve-7bf88d7d67-clv22\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:13.950767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.950744 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-model-cache\") pod \"test-model-79c24a-kserve-7bf88d7d67-clv22\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:13.952684 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.952648 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-dshm\") pod \"test-model-79c24a-kserve-7bf88d7d67-clv22\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:13.952949 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.952929 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50ced360-7983-4b5a-8b03-35714cd87a9c-tls-certs\") pod \"test-model-79c24a-kserve-7bf88d7d67-clv22\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:13.973694 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:13.973652 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khmjd\" (UniqueName: \"kubernetes.io/projected/50ced360-7983-4b5a-8b03-35714cd87a9c-kube-api-access-khmjd\") pod \"test-model-79c24a-kserve-7bf88d7d67-clv22\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:14.075709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:14.075352 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:14.370086 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:14.366198 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:14.391639 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:14.389013 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:14.658866 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:14.658833 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22"] Oct 30 21:29:14.659779 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:29:14.659738 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50ced360_7983_4b5a_8b03_35714cd87a9c.slice/crio-567fbc0028298e65749ddab15cc828733242f5b897cb0218b0a2d74b00290b9a WatchSource:0}: Error finding container 567fbc0028298e65749ddab15cc828733242f5b897cb0218b0a2d74b00290b9a: Status 404 returned error can't find the container with id 567fbc0028298e65749ddab15cc828733242f5b897cb0218b0a2d74b00290b9a Oct 30 21:29:14.830991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:14.830952 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" event={"ID":"50ced360-7983-4b5a-8b03-35714cd87a9c","Type":"ContainerStarted","Data":"bfdc50939933712eeb4cda35bb361990b417b01c89eb3c7cf222fc3adffb42ad"} Oct 30 21:29:14.831444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:14.830999 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" event={"ID":"50ced360-7983-4b5a-8b03-35714cd87a9c","Type":"ContainerStarted","Data":"567fbc0028298e65749ddab15cc828733242f5b897cb0218b0a2d74b00290b9a"} Oct 30 21:29:14.842522 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:14.842491 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:29:14.894006 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:14.893940 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" podStartSLOduration=1.8939208939999999 podStartE2EDuration="1.893920894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:14.889184842 +0000 UTC m=+1742.010258590" watchObservedRunningTime="2026-10-30 21:29:14.893920894 +0000 UTC m=+1742.014994643" Oct 30 21:29:15.090057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:15.078133 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:15.358225 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:15.358127 4926 scope.go:122] "RemoveContainer" containerID="2cd0fce63212f84feb0ee8aa9685a3eb562d9830d29f2cfd8323f3b7df046ec6" Oct 30 21:29:15.373612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:15.373584 4926 scope.go:122] "RemoveContainer" containerID="fc2880c7cb5096c5ab5141bd37cbcf6c20fbc417c8b2a44761fb45c11b886d25" Oct 30 21:29:15.387367 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:15.387333 4926 scope.go:122] "RemoveContainer" containerID="17d8f8263b8c94e9002ca4edf4792c100c0dabd4e6f10b6bf5c3411699ef84b8" Oct 30 21:29:20.259056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:20.253697 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:29:24.076111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:24.076067 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:24.095541 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:24.095507 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:24.959529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:24.959444 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:29:37.490855 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:29:37.490807 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:29:51.147111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.147073 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp"] Oct 30 21:29:51.153013 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.152989 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:29:51.178450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.178418 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3b28bc\"/\"openshift-service-ca.crt\"" Oct 30 21:29:51.178450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.178418 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3b28bc\"/\"kube-root-ca.crt\"" Oct 30 21:29:51.178654 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.178579 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3b28bc\"/\"default-dockercfg-xtf8l\"" Oct 30 21:29:51.178654 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.178589 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3b28bc\"/\"test-model-3b28bc-kserve-self-signed-certs\"" Oct 30 21:29:51.184753 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.184720 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp"] Oct 30 21:29:51.208975 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.208939 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-model-cache\") pod \"test-model-3b28bc-kserve-65dbc5c95-vcppp\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:29:51.209185 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.209163 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-dshm\") pod \"test-model-3b28bc-kserve-65dbc5c95-vcppp\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:29:51.209238 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.209225 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-home\") pod \"test-model-3b28bc-kserve-65dbc5c95-vcppp\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:29:51.209276 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.209257 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-tmp-dir\") pod \"test-model-3b28bc-kserve-65dbc5c95-vcppp\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:29:51.209391 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.209377 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09e24087-be45-4f70-bdc0-5af059800182-tls-certs\") pod \"test-model-3b28bc-kserve-65dbc5c95-vcppp\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:29:51.209467 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.209453 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55ngd\" (UniqueName: \"kubernetes.io/projected/09e24087-be45-4f70-bdc0-5af059800182-kube-api-access-55ngd\") pod \"test-model-3b28bc-kserve-65dbc5c95-vcppp\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:29:51.311358 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.311321 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-home\") pod \"test-model-3b28bc-kserve-65dbc5c95-vcppp\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:29:51.311568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.311366 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-tmp-dir\") pod \"test-model-3b28bc-kserve-65dbc5c95-vcppp\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:29:51.311568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.311481 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09e24087-be45-4f70-bdc0-5af059800182-tls-certs\") pod \"test-model-3b28bc-kserve-65dbc5c95-vcppp\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:29:51.311712 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.311572 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55ngd\" (UniqueName: \"kubernetes.io/projected/09e24087-be45-4f70-bdc0-5af059800182-kube-api-access-55ngd\") pod \"test-model-3b28bc-kserve-65dbc5c95-vcppp\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:29:51.311769 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.311741 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-model-cache\") pod \"test-model-3b28bc-kserve-65dbc5c95-vcppp\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:29:51.311861 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.311841 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-home\") pod \"test-model-3b28bc-kserve-65dbc5c95-vcppp\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:29:51.311926 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.311882 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-tmp-dir\") pod \"test-model-3b28bc-kserve-65dbc5c95-vcppp\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:29:51.311926 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.311917 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-dshm\") pod \"test-model-3b28bc-kserve-65dbc5c95-vcppp\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:29:51.312227 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.312206 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-model-cache\") pod \"test-model-3b28bc-kserve-65dbc5c95-vcppp\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:29:51.314893 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.314856 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-dshm\") pod \"test-model-3b28bc-kserve-65dbc5c95-vcppp\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:29:51.317236 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.317214 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09e24087-be45-4f70-bdc0-5af059800182-tls-certs\") pod \"test-model-3b28bc-kserve-65dbc5c95-vcppp\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:29:51.331502 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.331462 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55ngd\" (UniqueName: \"kubernetes.io/projected/09e24087-be45-4f70-bdc0-5af059800182-kube-api-access-55ngd\") pod \"test-model-3b28bc-kserve-65dbc5c95-vcppp\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:29:51.471372 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:51.471335 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:29:52.042781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:52.042756 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp"] Oct 30 21:29:52.043701 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:29:52.043674 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09e24087_be45_4f70_bdc0_5af059800182.slice/crio-4b3d1ce8f461b5a9dc80d13e42af6aff82435d7efa40bdaea30e0725acdbc0a9 WatchSource:0}: Error finding container 4b3d1ce8f461b5a9dc80d13e42af6aff82435d7efa40bdaea30e0725acdbc0a9: Status 404 returned error can't find the container with id 4b3d1ce8f461b5a9dc80d13e42af6aff82435d7efa40bdaea30e0725acdbc0a9 Oct 30 21:29:52.297279 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:52.295587 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" event={"ID":"09e24087-be45-4f70-bdc0-5af059800182","Type":"ContainerStarted","Data":"de180d1c0b637d03718fcaa34899bdc9578d2aed237ec1bfbcc3b0963e943d2a"} Oct 30 21:29:52.297279 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:52.295641 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" event={"ID":"09e24087-be45-4f70-bdc0-5af059800182","Type":"ContainerStarted","Data":"4b3d1ce8f461b5a9dc80d13e42af6aff82435d7efa40bdaea30e0725acdbc0a9"} Oct 30 21:29:52.350736 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:52.350671 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" podStartSLOduration=1.350655883 podStartE2EDuration="1.350655883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:29:52.346026691 +0000 UTC m=+1779.467100439" watchObservedRunningTime="2026-10-30 21:29:52.350655883 +0000 UTC m=+1779.471729629" Oct 30 21:29:52.483056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:52.478891 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:29:59.680378 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:59.680326 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s"] Oct 30 21:29:59.688125 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:59.686788 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:29:59.715190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:59.715157 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s"] Oct 30 21:29:59.721834 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:59.721537 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-79e911\"/\"test-model-79e911-kserve-self-signed-certs\"" Oct 30 21:29:59.721834 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:59.721636 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-79e911\"/\"openshift-service-ca.crt\"" Oct 30 21:29:59.721834 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:59.721668 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-79e911\"/\"kube-root-ca.crt\"" Oct 30 21:29:59.725371 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:59.725219 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-79e911\"/\"default-dockercfg-bjw2w\"" Oct 30 21:29:59.874441 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:59.874407 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-dshm\") pod \"test-model-79e911-kserve-7479c5cd75-8kw7s\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:29:59.874649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:59.874446 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2bb6ce-be54-4671-a234-759effcd695c-tls-certs\") pod \"test-model-79e911-kserve-7479c5cd75-8kw7s\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:29:59.874649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:59.874502 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-home\") pod \"test-model-79e911-kserve-7479c5cd75-8kw7s\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:29:59.874649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:59.874544 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-tmp-dir\") pod \"test-model-79e911-kserve-7479c5cd75-8kw7s\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:29:59.874828 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:59.874688 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-model-cache\") pod \"test-model-79e911-kserve-7479c5cd75-8kw7s\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:29:59.874828 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:59.874729 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mthq9\" (UniqueName: \"kubernetes.io/projected/9c2bb6ce-be54-4671-a234-759effcd695c-kube-api-access-mthq9\") pod \"test-model-79e911-kserve-7479c5cd75-8kw7s\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:29:59.981176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:59.979666 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-dshm\") pod \"test-model-79e911-kserve-7479c5cd75-8kw7s\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:29:59.981176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:59.979717 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2bb6ce-be54-4671-a234-759effcd695c-tls-certs\") pod \"test-model-79e911-kserve-7479c5cd75-8kw7s\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:29:59.981176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:59.979764 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-home\") pod \"test-model-79e911-kserve-7479c5cd75-8kw7s\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:29:59.981176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:59.979805 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-tmp-dir\") pod \"test-model-79e911-kserve-7479c5cd75-8kw7s\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:29:59.981176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:59.979928 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-model-cache\") pod \"test-model-79e911-kserve-7479c5cd75-8kw7s\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:29:59.981176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:59.979961 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mthq9\" (UniqueName: \"kubernetes.io/projected/9c2bb6ce-be54-4671-a234-759effcd695c-kube-api-access-mthq9\") pod \"test-model-79e911-kserve-7479c5cd75-8kw7s\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:29:59.981176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:59.981153 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-home\") pod \"test-model-79e911-kserve-7479c5cd75-8kw7s\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:29:59.987055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:59.981745 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-tmp-dir\") pod \"test-model-79e911-kserve-7479c5cd75-8kw7s\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:29:59.987055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:29:59.981974 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-model-cache\") pod \"test-model-79e911-kserve-7479c5cd75-8kw7s\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:30:00.022071 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.008614 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mthq9\" (UniqueName: \"kubernetes.io/projected/9c2bb6ce-be54-4671-a234-759effcd695c-kube-api-access-mthq9\") pod \"test-model-79e911-kserve-7479c5cd75-8kw7s\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:30:00.031065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.030591 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-dshm\") pod \"test-model-79e911-kserve-7479c5cd75-8kw7s\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:30:00.040065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.039732 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2bb6ce-be54-4671-a234-759effcd695c-tls-certs\") pod \"test-model-79e911-kserve-7479c5cd75-8kw7s\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:30:00.176445 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.176008 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4"] Oct 30 21:30:00.184328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.184301 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4" Oct 30 21:30:00.235983 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.235897 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:30:00.264555 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.263381 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv"] Oct 30 21:30:00.327391 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.327148 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:30:00.352695 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.345232 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2phj\" (UniqueName: \"kubernetes.io/projected/6d846cc6-bbdd-456a-a049-e9d99773172d-kube-api-access-c2phj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4\" (UID: \"6d846cc6-bbdd-456a-a049-e9d99773172d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4" Oct 30 21:30:00.352695 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.349021 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d846cc6-bbdd-456a-a049-e9d99773172d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4\" (UID: \"6d846cc6-bbdd-456a-a049-e9d99773172d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4" Oct 30 21:30:00.382717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.382681 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4"] Oct 30 21:30:00.385464 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.385433 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8"] Oct 30 21:30:00.391778 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.391748 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" Oct 30 21:30:00.418612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.414447 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj"] Oct 30 21:30:00.418612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.417595 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8" Oct 30 21:30:00.424963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.422572 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r"] Oct 30 21:30:00.424963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.424184 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj" Oct 30 21:30:00.431095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.429150 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc"] Oct 30 21:30:00.431095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.430262 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" Oct 30 21:30:00.440108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.439839 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv"] Oct 30 21:30:00.440108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.439880 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r"] Oct 30 21:30:00.440108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.439968 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc" Oct 30 21:30:00.443317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.443295 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9"] Oct 30 21:30:00.449011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.448886 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9" Oct 30 21:30:00.451840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.451812 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/731eb7a0-c9e9-41f9-b245-0fe7a00be673-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8\" (UID: \"731eb7a0-c9e9-41f9-b245-0fe7a00be673\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8" Oct 30 21:30:00.451959 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.451873 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2phj\" (UniqueName: \"kubernetes.io/projected/6d846cc6-bbdd-456a-a049-e9d99773172d-kube-api-access-c2phj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4\" (UID: \"6d846cc6-bbdd-456a-a049-e9d99773172d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4" Oct 30 21:30:00.452023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.452003 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shchh\" (UniqueName: \"kubernetes.io/projected/731eb7a0-c9e9-41f9-b245-0fe7a00be673-kube-api-access-shchh\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8\" (UID: \"731eb7a0-c9e9-41f9-b245-0fe7a00be673\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8" Oct 30 21:30:00.452262 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.452233 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r2l6\" (UniqueName: \"kubernetes.io/projected/c461be4b-e409-48fe-bcaf-6ba8e18ab312-kube-api-access-2r2l6\") pod \"maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj\" (UID: \"c461be4b-e409-48fe-bcaf-6ba8e18ab312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj" Oct 30 21:30:00.452351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.452336 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c461be4b-e409-48fe-bcaf-6ba8e18ab312-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj\" (UID: \"c461be4b-e409-48fe-bcaf-6ba8e18ab312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj" Oct 30 21:30:00.452413 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.452376 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d846cc6-bbdd-456a-a049-e9d99773172d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4\" (UID: \"6d846cc6-bbdd-456a-a049-e9d99773172d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4" Oct 30 21:30:00.452470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.452415 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8xhn\" (UniqueName: \"kubernetes.io/projected/495acf58-df75-4721-aaeb-fa3dcb9a3cd9-kube-api-access-t8xhn\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv\" (UID: \"495acf58-df75-4721-aaeb-fa3dcb9a3cd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" Oct 30 21:30:00.452470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.452448 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/495acf58-df75-4721-aaeb-fa3dcb9a3cd9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv\" (UID: \"495acf58-df75-4721-aaeb-fa3dcb9a3cd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" Oct 30 21:30:00.453583 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.453550 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d846cc6-bbdd-456a-a049-e9d99773172d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4\" (UID: \"6d846cc6-bbdd-456a-a049-e9d99773172d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4" Oct 30 21:30:00.454255 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.454232 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc"] Oct 30 21:30:00.463125 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.463097 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj"] Oct 30 21:30:00.476674 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.476634 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8"] Oct 30 21:30:00.485497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.485471 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg"] Oct 30 21:30:00.492228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.492167 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg" Oct 30 21:30:00.521708 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.520621 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2phj\" (UniqueName: \"kubernetes.io/projected/6d846cc6-bbdd-456a-a049-e9d99773172d-kube-api-access-c2phj\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4\" (UID: \"6d846cc6-bbdd-456a-a049-e9d99773172d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4" Oct 30 21:30:00.544990 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.544896 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg"] Oct 30 21:30:00.591727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.580790 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shchh\" (UniqueName: \"kubernetes.io/projected/731eb7a0-c9e9-41f9-b245-0fe7a00be673-kube-api-access-shchh\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8\" (UID: \"731eb7a0-c9e9-41f9-b245-0fe7a00be673\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8" Oct 30 21:30:00.591727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.580910 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde714d2-046b-443a-a5e6-f87c729e5ab0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9\" (UID: \"bde714d2-046b-443a-a5e6-f87c729e5ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9" Oct 30 21:30:00.591727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.581051 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55b8ddb-6b3e-4f51-8281-f3209c63d949-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-lmlbc\" (UID: \"d55b8ddb-6b3e-4f51-8281-f3209c63d949\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc" Oct 30 21:30:00.591727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.581129 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r2l6\" (UniqueName: \"kubernetes.io/projected/c461be4b-e409-48fe-bcaf-6ba8e18ab312-kube-api-access-2r2l6\") pod \"maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj\" (UID: \"c461be4b-e409-48fe-bcaf-6ba8e18ab312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj" Oct 30 21:30:00.591727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.581178 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5l69\" (UniqueName: \"kubernetes.io/projected/f5d36899-edf3-4bb0-a13b-39c36bf0ed68-kube-api-access-v5l69\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r\" (UID: \"f5d36899-edf3-4bb0-a13b-39c36bf0ed68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" Oct 30 21:30:00.591727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.581268 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d922621f-a5d6-4854-8f0e-ba0ee21ade8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg\" (UID: \"d922621f-a5d6-4854-8f0e-ba0ee21ade8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg" Oct 30 21:30:00.591727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.581322 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c461be4b-e409-48fe-bcaf-6ba8e18ab312-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj\" (UID: \"c461be4b-e409-48fe-bcaf-6ba8e18ab312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj" Oct 30 21:30:00.591727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.581379 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8xhn\" (UniqueName: \"kubernetes.io/projected/495acf58-df75-4721-aaeb-fa3dcb9a3cd9-kube-api-access-t8xhn\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv\" (UID: \"495acf58-df75-4721-aaeb-fa3dcb9a3cd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" Oct 30 21:30:00.591727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.581414 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/495acf58-df75-4721-aaeb-fa3dcb9a3cd9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv\" (UID: \"495acf58-df75-4721-aaeb-fa3dcb9a3cd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" Oct 30 21:30:00.591727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.581456 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d36899-edf3-4bb0-a13b-39c36bf0ed68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r\" (UID: \"f5d36899-edf3-4bb0-a13b-39c36bf0ed68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" Oct 30 21:30:00.591727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.581488 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcrkp\" (UniqueName: \"kubernetes.io/projected/d55b8ddb-6b3e-4f51-8281-f3209c63d949-kube-api-access-fcrkp\") pod \"maas-api-key-cleanup-29889930-lmlbc\" (UID: \"d55b8ddb-6b3e-4f51-8281-f3209c63d949\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc" Oct 30 21:30:00.591727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.581518 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqmlh\" (UniqueName: \"kubernetes.io/projected/bde714d2-046b-443a-a5e6-f87c729e5ab0-kube-api-access-qqmlh\") pod \"maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9\" (UID: \"bde714d2-046b-443a-a5e6-f87c729e5ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9" Oct 30 21:30:00.591727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.581570 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/731eb7a0-c9e9-41f9-b245-0fe7a00be673-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8\" (UID: \"731eb7a0-c9e9-41f9-b245-0fe7a00be673\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8" Oct 30 21:30:00.591727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.581687 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5kkp\" (UniqueName: \"kubernetes.io/projected/d922621f-a5d6-4854-8f0e-ba0ee21ade8e-kube-api-access-x5kkp\") pod \"maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg\" (UID: \"d922621f-a5d6-4854-8f0e-ba0ee21ade8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg" Oct 30 21:30:00.591727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.583359 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c461be4b-e409-48fe-bcaf-6ba8e18ab312-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj\" (UID: \"c461be4b-e409-48fe-bcaf-6ba8e18ab312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj" Oct 30 21:30:00.591727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.584294 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/495acf58-df75-4721-aaeb-fa3dcb9a3cd9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv\" (UID: \"495acf58-df75-4721-aaeb-fa3dcb9a3cd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" Oct 30 21:30:00.591727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.585094 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/731eb7a0-c9e9-41f9-b245-0fe7a00be673-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8\" (UID: \"731eb7a0-c9e9-41f9-b245-0fe7a00be673\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8" Oct 30 21:30:00.625486 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.624077 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9"] Oct 30 21:30:00.632858 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.632804 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8xhn\" (UniqueName: \"kubernetes.io/projected/495acf58-df75-4721-aaeb-fa3dcb9a3cd9-kube-api-access-t8xhn\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv\" (UID: \"495acf58-df75-4721-aaeb-fa3dcb9a3cd9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" Oct 30 21:30:00.635175 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.635144 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r2l6\" (UniqueName: \"kubernetes.io/projected/c461be4b-e409-48fe-bcaf-6ba8e18ab312-kube-api-access-2r2l6\") pod \"maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj\" (UID: \"c461be4b-e409-48fe-bcaf-6ba8e18ab312\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj" Oct 30 21:30:00.644249 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.640498 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6"] Oct 30 21:30:00.644249 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.641524 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shchh\" (UniqueName: \"kubernetes.io/projected/731eb7a0-c9e9-41f9-b245-0fe7a00be673-kube-api-access-shchh\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8\" (UID: \"731eb7a0-c9e9-41f9-b245-0fe7a00be673\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8" Oct 30 21:30:00.649984 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.649374 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6" Oct 30 21:30:00.649984 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.649457 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28"] Oct 30 21:30:00.658380 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.658069 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" Oct 30 21:30:00.659335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.659275 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9"] Oct 30 21:30:00.667399 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.667263 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6"] Oct 30 21:30:00.667399 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.667378 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9" Oct 30 21:30:00.671540 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.671372 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4" Oct 30 21:30:00.683666 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.683634 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5l69\" (UniqueName: \"kubernetes.io/projected/f5d36899-edf3-4bb0-a13b-39c36bf0ed68-kube-api-access-v5l69\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r\" (UID: \"f5d36899-edf3-4bb0-a13b-39c36bf0ed68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" Oct 30 21:30:00.684206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.683689 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d922621f-a5d6-4854-8f0e-ba0ee21ade8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg\" (UID: \"d922621f-a5d6-4854-8f0e-ba0ee21ade8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg" Oct 30 21:30:00.684206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.683778 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d36899-edf3-4bb0-a13b-39c36bf0ed68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r\" (UID: \"f5d36899-edf3-4bb0-a13b-39c36bf0ed68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" Oct 30 21:30:00.684206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.683808 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcrkp\" (UniqueName: \"kubernetes.io/projected/d55b8ddb-6b3e-4f51-8281-f3209c63d949-kube-api-access-fcrkp\") pod \"maas-api-key-cleanup-29889930-lmlbc\" (UID: \"d55b8ddb-6b3e-4f51-8281-f3209c63d949\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc" Oct 30 21:30:00.684206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.683833 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqmlh\" (UniqueName: \"kubernetes.io/projected/bde714d2-046b-443a-a5e6-f87c729e5ab0-kube-api-access-qqmlh\") pod \"maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9\" (UID: \"bde714d2-046b-443a-a5e6-f87c729e5ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9" Oct 30 21:30:00.684206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.683943 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5kkp\" (UniqueName: \"kubernetes.io/projected/d922621f-a5d6-4854-8f0e-ba0ee21ade8e-kube-api-access-x5kkp\") pod \"maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg\" (UID: \"d922621f-a5d6-4854-8f0e-ba0ee21ade8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg" Oct 30 21:30:00.684206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.684097 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde714d2-046b-443a-a5e6-f87c729e5ab0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9\" (UID: \"bde714d2-046b-443a-a5e6-f87c729e5ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9" Oct 30 21:30:00.684607 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.684214 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55b8ddb-6b3e-4f51-8281-f3209c63d949-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-lmlbc\" (UID: \"d55b8ddb-6b3e-4f51-8281-f3209c63d949\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc" Oct 30 21:30:00.685111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.685084 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55b8ddb-6b3e-4f51-8281-f3209c63d949-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889930-lmlbc\" (UID: \"d55b8ddb-6b3e-4f51-8281-f3209c63d949\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc" Oct 30 21:30:00.685659 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.685631 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6"] Oct 30 21:30:00.690258 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.687345 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d36899-edf3-4bb0-a13b-39c36bf0ed68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r\" (UID: \"f5d36899-edf3-4bb0-a13b-39c36bf0ed68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" Oct 30 21:30:00.690258 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.690218 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde714d2-046b-443a-a5e6-f87c729e5ab0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9\" (UID: \"bde714d2-046b-443a-a5e6-f87c729e5ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9" Oct 30 21:30:00.690432 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.690407 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d922621f-a5d6-4854-8f0e-ba0ee21ade8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg\" (UID: \"d922621f-a5d6-4854-8f0e-ba0ee21ade8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg" Oct 30 21:30:00.708050 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.702726 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" Oct 30 21:30:00.728219 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.728010 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" Oct 30 21:30:00.731122 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.730647 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcrkp\" (UniqueName: \"kubernetes.io/projected/d55b8ddb-6b3e-4f51-8281-f3209c63d949-kube-api-access-fcrkp\") pod \"maas-api-key-cleanup-29889930-lmlbc\" (UID: \"d55b8ddb-6b3e-4f51-8281-f3209c63d949\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc" Oct 30 21:30:00.739269 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.737804 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28"] Oct 30 21:30:00.743911 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.743150 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq"] Oct 30 21:30:00.743911 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.743807 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5kkp\" (UniqueName: \"kubernetes.io/projected/d922621f-a5d6-4854-8f0e-ba0ee21ade8e-kube-api-access-x5kkp\") pod \"maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg\" (UID: \"d922621f-a5d6-4854-8f0e-ba0ee21ade8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg" Oct 30 21:30:00.748076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.744820 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5l69\" (UniqueName: \"kubernetes.io/projected/f5d36899-edf3-4bb0-a13b-39c36bf0ed68-kube-api-access-v5l69\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r\" (UID: \"f5d36899-edf3-4bb0-a13b-39c36bf0ed68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" Oct 30 21:30:00.767871 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.767838 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9"] Oct 30 21:30:00.768020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.767905 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6"] Oct 30 21:30:00.768020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.768014 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq" Oct 30 21:30:00.779384 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.775006 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqmlh\" (UniqueName: \"kubernetes.io/projected/bde714d2-046b-443a-a5e6-f87c729e5ab0-kube-api-access-qqmlh\") pod \"maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9\" (UID: \"bde714d2-046b-443a-a5e6-f87c729e5ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9" Oct 30 21:30:00.779384 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.775106 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4"] Oct 30 21:30:00.786357 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.786320 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd7f7c91-0516-476c-afc8-94130b2e3439-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9\" (UID: \"fd7f7c91-0516-476c-afc8-94130b2e3439\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9" Oct 30 21:30:00.786555 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.786533 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znkj9\" (UniqueName: \"kubernetes.io/projected/0fe4049a-063d-4990-8524-88c95c2d8862-kube-api-access-znkj9\") pod \"maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28\" (UID: \"0fe4049a-063d-4990-8524-88c95c2d8862\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" Oct 30 21:30:00.786632 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.786592 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/810d266f-62a8-4b0f-a146-89ed2e41df99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6\" (UID: \"810d266f-62a8-4b0f-a146-89ed2e41df99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6" Oct 30 21:30:00.786699 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.786630 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfcmv\" (UniqueName: \"kubernetes.io/projected/63543c7d-5e17-4248-82f2-b5b947e7aa4c-kube-api-access-rfcmv\") pod \"maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6\" (UID: \"63543c7d-5e17-4248-82f2-b5b947e7aa4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" Oct 30 21:30:00.786699 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.786679 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63543c7d-5e17-4248-82f2-b5b947e7aa4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6\" (UID: \"63543c7d-5e17-4248-82f2-b5b947e7aa4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" Oct 30 21:30:00.786810 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.786794 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmhvz\" (UniqueName: \"kubernetes.io/projected/810d266f-62a8-4b0f-a146-89ed2e41df99-kube-api-access-wmhvz\") pod \"maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6\" (UID: \"810d266f-62a8-4b0f-a146-89ed2e41df99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6" Oct 30 21:30:00.786874 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.786826 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwtsw\" (UniqueName: \"kubernetes.io/projected/fd7f7c91-0516-476c-afc8-94130b2e3439-kube-api-access-jwtsw\") pod \"maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9\" (UID: \"fd7f7c91-0516-476c-afc8-94130b2e3439\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9" Oct 30 21:30:00.786946 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.786927 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fe4049a-063d-4990-8524-88c95c2d8862-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28\" (UID: \"0fe4049a-063d-4990-8524-88c95c2d8862\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" Oct 30 21:30:00.794460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.794270 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj" Oct 30 21:30:00.795329 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.795275 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8" Oct 30 21:30:00.796990 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.796966 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq"] Oct 30 21:30:00.797347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.797262 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4" Oct 30 21:30:00.807782 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.807655 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4"] Oct 30 21:30:00.813650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.813508 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" Oct 30 21:30:00.831261 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.831227 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc" Oct 30 21:30:00.858180 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.858121 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s"] Oct 30 21:30:00.892209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.892145 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/810d266f-62a8-4b0f-a146-89ed2e41df99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6\" (UID: \"810d266f-62a8-4b0f-a146-89ed2e41df99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6" Oct 30 21:30:00.892209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.892202 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfcmv\" (UniqueName: \"kubernetes.io/projected/63543c7d-5e17-4248-82f2-b5b947e7aa4c-kube-api-access-rfcmv\") pod \"maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6\" (UID: \"63543c7d-5e17-4248-82f2-b5b947e7aa4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" Oct 30 21:30:00.892423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.892258 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63543c7d-5e17-4248-82f2-b5b947e7aa4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6\" (UID: \"63543c7d-5e17-4248-82f2-b5b947e7aa4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" Oct 30 21:30:00.892423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.892385 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmhvz\" (UniqueName: \"kubernetes.io/projected/810d266f-62a8-4b0f-a146-89ed2e41df99-kube-api-access-wmhvz\") pod \"maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6\" (UID: \"810d266f-62a8-4b0f-a146-89ed2e41df99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6" Oct 30 21:30:00.892591 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.892418 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwtsw\" (UniqueName: \"kubernetes.io/projected/fd7f7c91-0516-476c-afc8-94130b2e3439-kube-api-access-jwtsw\") pod \"maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9\" (UID: \"fd7f7c91-0516-476c-afc8-94130b2e3439\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9" Oct 30 21:30:00.892591 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.892465 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17fd537-658d-4c4f-9aeb-a763d184c364-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4\" (UID: \"f17fd537-658d-4c4f-9aeb-a763d184c364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4" Oct 30 21:30:00.892591 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.892556 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fe4049a-063d-4990-8524-88c95c2d8862-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28\" (UID: \"0fe4049a-063d-4990-8524-88c95c2d8862\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" Oct 30 21:30:00.892749 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.892722 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd7f7c91-0516-476c-afc8-94130b2e3439-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9\" (UID: \"fd7f7c91-0516-476c-afc8-94130b2e3439\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9" Oct 30 21:30:00.892808 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.892774 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns466\" (UniqueName: \"kubernetes.io/projected/fce0f54b-5f27-4637-8b5c-d6f2298873bb-kube-api-access-ns466\") pod \"maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq\" (UID: \"fce0f54b-5f27-4637-8b5c-d6f2298873bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq" Oct 30 21:30:00.892862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.892826 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b5f5\" (UniqueName: \"kubernetes.io/projected/f17fd537-658d-4c4f-9aeb-a763d184c364-kube-api-access-2b5f5\") pod \"maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4\" (UID: \"f17fd537-658d-4c4f-9aeb-a763d184c364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4" Oct 30 21:30:00.892913 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.892877 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce0f54b-5f27-4637-8b5c-d6f2298873bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq\" (UID: \"fce0f54b-5f27-4637-8b5c-d6f2298873bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq" Oct 30 21:30:00.892971 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.892946 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znkj9\" (UniqueName: \"kubernetes.io/projected/0fe4049a-063d-4990-8524-88c95c2d8862-kube-api-access-znkj9\") pod \"maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28\" (UID: \"0fe4049a-063d-4990-8524-88c95c2d8862\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" Oct 30 21:30:00.894288 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.894260 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/810d266f-62a8-4b0f-a146-89ed2e41df99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6\" (UID: \"810d266f-62a8-4b0f-a146-89ed2e41df99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6" Oct 30 21:30:00.895128 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.895021 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fe4049a-063d-4990-8524-88c95c2d8862-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28\" (UID: \"0fe4049a-063d-4990-8524-88c95c2d8862\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" Oct 30 21:30:00.928588 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.903967 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63543c7d-5e17-4248-82f2-b5b947e7aa4c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6\" (UID: \"63543c7d-5e17-4248-82f2-b5b947e7aa4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" Oct 30 21:30:00.928588 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.915625 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9" Oct 30 21:30:00.928588 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.921186 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd7f7c91-0516-476c-afc8-94130b2e3439-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9\" (UID: \"fd7f7c91-0516-476c-afc8-94130b2e3439\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9" Oct 30 21:30:00.928588 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.924821 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znkj9\" (UniqueName: \"kubernetes.io/projected/0fe4049a-063d-4990-8524-88c95c2d8862-kube-api-access-znkj9\") pod \"maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28\" (UID: \"0fe4049a-063d-4990-8524-88c95c2d8862\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" Oct 30 21:30:00.928588 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.926763 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfcmv\" (UniqueName: \"kubernetes.io/projected/63543c7d-5e17-4248-82f2-b5b947e7aa4c-kube-api-access-rfcmv\") pod \"maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6\" (UID: \"63543c7d-5e17-4248-82f2-b5b947e7aa4c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" Oct 30 21:30:00.938786 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.938571 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg" Oct 30 21:30:00.946154 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.944946 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmhvz\" (UniqueName: \"kubernetes.io/projected/810d266f-62a8-4b0f-a146-89ed2e41df99-kube-api-access-wmhvz\") pod \"maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6\" (UID: \"810d266f-62a8-4b0f-a146-89ed2e41df99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6" Oct 30 21:30:00.947002 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.946978 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwtsw\" (UniqueName: \"kubernetes.io/projected/fd7f7c91-0516-476c-afc8-94130b2e3439-kube-api-access-jwtsw\") pod \"maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9\" (UID: \"fd7f7c91-0516-476c-afc8-94130b2e3439\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9" Oct 30 21:30:00.976266 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.975432 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6" Oct 30 21:30:00.995330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.994581 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17fd537-658d-4c4f-9aeb-a763d184c364-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4\" (UID: \"f17fd537-658d-4c4f-9aeb-a763d184c364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4" Oct 30 21:30:00.995330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.994826 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns466\" (UniqueName: \"kubernetes.io/projected/fce0f54b-5f27-4637-8b5c-d6f2298873bb-kube-api-access-ns466\") pod \"maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq\" (UID: \"fce0f54b-5f27-4637-8b5c-d6f2298873bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq" Oct 30 21:30:00.995330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.994878 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b5f5\" (UniqueName: \"kubernetes.io/projected/f17fd537-658d-4c4f-9aeb-a763d184c364-kube-api-access-2b5f5\") pod \"maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4\" (UID: \"f17fd537-658d-4c4f-9aeb-a763d184c364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4" Oct 30 21:30:00.995330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.994923 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce0f54b-5f27-4637-8b5c-d6f2298873bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq\" (UID: \"fce0f54b-5f27-4637-8b5c-d6f2298873bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq" Oct 30 21:30:00.997636 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.996368 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce0f54b-5f27-4637-8b5c-d6f2298873bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq\" (UID: \"fce0f54b-5f27-4637-8b5c-d6f2298873bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq" Oct 30 21:30:00.997636 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:00.997129 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17fd537-658d-4c4f-9aeb-a763d184c364-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4\" (UID: \"f17fd537-658d-4c4f-9aeb-a763d184c364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4" Oct 30 21:30:01.002457 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:01.002196 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" Oct 30 21:30:01.025056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:01.024991 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b5f5\" (UniqueName: \"kubernetes.io/projected/f17fd537-658d-4c4f-9aeb-a763d184c364-kube-api-access-2b5f5\") pod \"maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4\" (UID: \"f17fd537-658d-4c4f-9aeb-a763d184c364\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4" Oct 30 21:30:01.027286 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:01.027263 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9" Oct 30 21:30:01.028300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:01.028082 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns466\" (UniqueName: \"kubernetes.io/projected/fce0f54b-5f27-4637-8b5c-d6f2298873bb-kube-api-access-ns466\") pod \"maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq\" (UID: \"fce0f54b-5f27-4637-8b5c-d6f2298873bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq" Oct 30 21:30:01.105081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:01.105043 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" Oct 30 21:30:01.121295 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:01.121206 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4"] Oct 30 21:30:01.143592 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:01.143560 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq" Oct 30 21:30:01.222054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:01.221828 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4" Oct 30 21:30:01.325726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:01.310763 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv"] Oct 30 21:30:01.480840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:01.478765 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:30:01.533906 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:01.532555 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:30:01.535159 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:01.535092 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj"] Oct 30 21:30:01.535357 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:01.535330 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" event={"ID":"9c2bb6ce-be54-4671-a234-759effcd695c","Type":"ContainerStarted","Data":"5788041a8b152acfb1674a710814271f42981f996f1638310bad74402cc3508e"} Oct 30 21:30:01.535509 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:01.535487 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" event={"ID":"9c2bb6ce-be54-4671-a234-759effcd695c","Type":"ContainerStarted","Data":"49b0ab9cbf0670f90749cd63e6232f2ce2f39a269000a12f3131d311b5c9d08e"} Oct 30 21:30:01.537982 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:01.537901 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4" event={"ID":"6d846cc6-bbdd-456a-a049-e9d99773172d","Type":"ContainerStarted","Data":"beba957622d7e596338ddfe08ac2f7f44086ce7f8488f8f294f460905167c87e"} Oct 30 21:30:01.541105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:01.541086 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" event={"ID":"495acf58-df75-4721-aaeb-fa3dcb9a3cd9","Type":"ContainerStarted","Data":"5bc106f185b03c4e6b7d2dfcbd8b0d75351ebe3e1d3e4aaeb6b64ed2fc9e6415"} Oct 30 21:30:01.552829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:01.552799 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8"] Oct 30 21:30:01.686839 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:30:01.686768 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55b8ddb_6b3e_4f51_8281_f3209c63d949.slice/crio-d4f5ff07ee094cbdbd85243b2983b8c22a1230629566fa37223174a7fd399be5 WatchSource:0}: Error finding container d4f5ff07ee094cbdbd85243b2983b8c22a1230629566fa37223174a7fd399be5: Status 404 returned error can't find the container with id d4f5ff07ee094cbdbd85243b2983b8c22a1230629566fa37223174a7fd399be5 Oct 30 21:30:01.698845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:01.689075 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc"] Oct 30 21:30:01.698845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:01.692538 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" podStartSLOduration=2.692515098 podStartE2EDuration="2.692515098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:01.682980774 +0000 UTC m=+1788.804054528" watchObservedRunningTime="2026-10-30 21:30:01.692515098 +0000 UTC m=+1788.813588846" Oct 30 21:30:01.731290 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:01.731256 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r"] Oct 30 21:30:01.907053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:01.904672 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9"] Oct 30 21:30:02.029996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.029964 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg"] Oct 30 21:30:02.042572 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.042545 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28"] Oct 30 21:30:02.045546 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:30:02.045509 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe4049a_063d_4990_8524_88c95c2d8862.slice/crio-198932983fa93708461213ec05eeae28da3e2b063d973cd0f95bfda4fe093910 WatchSource:0}: Error finding container 198932983fa93708461213ec05eeae28da3e2b063d973cd0f95bfda4fe093910: Status 404 returned error can't find the container with id 198932983fa93708461213ec05eeae28da3e2b063d973cd0f95bfda4fe093910 Oct 30 21:30:02.094249 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.094216 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq"] Oct 30 21:30:02.123313 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.120861 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9"] Oct 30 21:30:02.134044 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.133998 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6"] Oct 30 21:30:02.196986 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.196953 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6"] Oct 30 21:30:02.201861 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.201835 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4"] Oct 30 21:30:02.207001 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:30:02.206722 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810d266f_62a8_4b0f_a146_89ed2e41df99.slice/crio-c9168aaa1d70354b1794e65bdf184fd3e419dbcc9488576317e91aeeae403d2f WatchSource:0}: Error finding container c9168aaa1d70354b1794e65bdf184fd3e419dbcc9488576317e91aeeae403d2f: Status 404 returned error can't find the container with id c9168aaa1d70354b1794e65bdf184fd3e419dbcc9488576317e91aeeae403d2f Oct 30 21:30:02.225267 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:30:02.225232 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf17fd537_658d_4c4f_9aeb_a763d184c364.slice/crio-dce1b494793709bab0b66803cc37378cba9ff12bdcdeb098444a27e3fdf71a0d WatchSource:0}: Error finding container dce1b494793709bab0b66803cc37378cba9ff12bdcdeb098444a27e3fdf71a0d: Status 404 returned error can't find the container with id dce1b494793709bab0b66803cc37378cba9ff12bdcdeb098444a27e3fdf71a0d Oct 30 21:30:02.336163 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.336125 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:30:02.494751 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.494564 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:30:02.553760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.553285 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" event={"ID":"495acf58-df75-4721-aaeb-fa3dcb9a3cd9","Type":"ContainerStarted","Data":"7a4dc82a11f21b7109031a380be79af07d50c1f1616821d6ed9020473335ad3f"} Oct 30 21:30:02.555761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.555686 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" event={"ID":"f5d36899-edf3-4bb0-a13b-39c36bf0ed68","Type":"ContainerStarted","Data":"2c8e8f8c309c509bb754789757ea90cce7ac456ed4256bd06f83947c6597df2e"} Oct 30 21:30:02.560277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.560198 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" event={"ID":"0fe4049a-063d-4990-8524-88c95c2d8862","Type":"ContainerStarted","Data":"198932983fa93708461213ec05eeae28da3e2b063d973cd0f95bfda4fe093910"} Oct 30 21:30:02.562330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.562259 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4" event={"ID":"f17fd537-658d-4c4f-9aeb-a763d184c364","Type":"ContainerStarted","Data":"dce1b494793709bab0b66803cc37378cba9ff12bdcdeb098444a27e3fdf71a0d"} Oct 30 21:30:02.565529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.564535 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc" event={"ID":"d55b8ddb-6b3e-4f51-8281-f3209c63d949","Type":"ContainerStarted","Data":"25bfe40aca640e34ed8078248d04bcdfd28dcf27306b0b1c283f9ec234573f1a"} Oct 30 21:30:02.565529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.564581 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc" event={"ID":"d55b8ddb-6b3e-4f51-8281-f3209c63d949","Type":"ContainerStarted","Data":"d4f5ff07ee094cbdbd85243b2983b8c22a1230629566fa37223174a7fd399be5"} Oct 30 21:30:02.572266 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.572244 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj" event={"ID":"c461be4b-e409-48fe-bcaf-6ba8e18ab312","Type":"ContainerStarted","Data":"4e9db721d59e6f0af872ce51a59d73d93d9b46511b397318b4e591a6734363ad"} Oct 30 21:30:02.572417 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.572404 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj" event={"ID":"c461be4b-e409-48fe-bcaf-6ba8e18ab312","Type":"ContainerStarted","Data":"a1c73588199c8d9a91a620bd9f93dcbdefea0191cb2400349d66b2088f842059"} Oct 30 21:30:02.575369 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.575347 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9" event={"ID":"fd7f7c91-0516-476c-afc8-94130b2e3439","Type":"ContainerStarted","Data":"3aa135ef51b2c3b573a3aea991aa1520e02af382f859f08a237b6bbdb2c42066"} Oct 30 21:30:02.578480 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.577562 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4" event={"ID":"6d846cc6-bbdd-456a-a049-e9d99773172d","Type":"ContainerStarted","Data":"2d94a5e0e43a04fd006499ea0987d76ff99c8301c44103e16e7c580ae8fa3626"} Oct 30 21:30:02.580152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.580103 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" event={"ID":"63543c7d-5e17-4248-82f2-b5b947e7aa4c","Type":"ContainerStarted","Data":"acef094367c547b9ac9b98afd23cd132ef0b48f0193bb5caadd3c30fa11817dc"} Oct 30 21:30:02.581644 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.581590 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6" event={"ID":"810d266f-62a8-4b0f-a146-89ed2e41df99","Type":"ContainerStarted","Data":"c9168aaa1d70354b1794e65bdf184fd3e419dbcc9488576317e91aeeae403d2f"} Oct 30 21:30:02.587744 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.587697 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9" event={"ID":"bde714d2-046b-443a-a5e6-f87c729e5ab0","Type":"ContainerStarted","Data":"a7a12e7ef51f35b8f3c49dec9f0a8b5bd656190280be3c92e913b61690837280"} Oct 30 21:30:02.591351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.591314 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8" event={"ID":"731eb7a0-c9e9-41f9-b245-0fe7a00be673","Type":"ContainerStarted","Data":"5906c65d92616f8ac519864d6fa5b03a31f202bdbf89b028fa3b006cf0f16816"} Oct 30 21:30:02.591834 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.591782 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8" event={"ID":"731eb7a0-c9e9-41f9-b245-0fe7a00be673","Type":"ContainerStarted","Data":"0cbe6fb2c2efe739f1fcab313a13f23810be8d057c51317c56010ea12a0b39de"} Oct 30 21:30:02.597277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.597238 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg" event={"ID":"d922621f-a5d6-4854-8f0e-ba0ee21ade8e","Type":"ContainerStarted","Data":"f4383edc38fc9eb9e989e08c4cad314e4d9b31c9e7a0785bc92782eae89d4847"} Oct 30 21:30:02.608197 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.608167 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq" event={"ID":"fce0f54b-5f27-4637-8b5c-d6f2298873bb","Type":"ContainerStarted","Data":"cd761a62a9bc014e15361d5734b51baa474c8d30c2457b7d9e28f7442498ebe7"} Oct 30 21:30:02.619089 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.618999 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" podStartSLOduration=2.618978178 podStartE2EDuration="2.618978178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:02.599929608 +0000 UTC m=+1789.721003355" watchObservedRunningTime="2026-10-30 21:30:02.618978178 +0000 UTC m=+1789.740051925" Oct 30 21:30:02.642912 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.642857 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc" podStartSLOduration=2.642842823 podStartE2EDuration="2.642842823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:02.638518947 +0000 UTC m=+1789.759592694" watchObservedRunningTime="2026-10-30 21:30:02.642842823 +0000 UTC m=+1789.763916568" Oct 30 21:30:02.677621 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.677152 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8" podStartSLOduration=2.677136982 podStartE2EDuration="2.677136982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:02.675809336 +0000 UTC m=+1789.796883095" watchObservedRunningTime="2026-10-30 21:30:02.677136982 +0000 UTC m=+1789.798210764" Oct 30 21:30:02.721894 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.721834 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4" podStartSLOduration=2.721817035 podStartE2EDuration="2.721817035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:02.716710267 +0000 UTC m=+1789.837784010" watchObservedRunningTime="2026-10-30 21:30:02.721817035 +0000 UTC m=+1789.842890783" Oct 30 21:30:02.784927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:02.784857 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj" podStartSLOduration=2.7848362 podStartE2EDuration="2.7848362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:02.755166294 +0000 UTC m=+1789.876240050" watchObservedRunningTime="2026-10-30 21:30:02.7848362 +0000 UTC m=+1789.905909949" Oct 30 21:30:03.620985 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.620944 4926 generic.go:356] "Generic (PLEG): container finished" podID="c461be4b-e409-48fe-bcaf-6ba8e18ab312" containerID="4e9db721d59e6f0af872ce51a59d73d93d9b46511b397318b4e591a6734363ad" exitCode=60 Oct 30 21:30:03.621256 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.621042 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj" event={"ID":"c461be4b-e409-48fe-bcaf-6ba8e18ab312","Type":"ContainerDied","Data":"4e9db721d59e6f0af872ce51a59d73d93d9b46511b397318b4e591a6734363ad"} Oct 30 21:30:03.621400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.621382 4926 scope.go:122] "RemoveContainer" containerID="4e9db721d59e6f0af872ce51a59d73d93d9b46511b397318b4e591a6734363ad" Oct 30 21:30:03.623014 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.622993 4926 generic.go:356] "Generic (PLEG): container finished" podID="fd7f7c91-0516-476c-afc8-94130b2e3439" containerID="0e0cf6be47b357d3f80da917af1dc9b9fab0fd611e94ef3f88daf370c599f61b" exitCode=60 Oct 30 21:30:03.623164 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.623062 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9" event={"ID":"fd7f7c91-0516-476c-afc8-94130b2e3439","Type":"ContainerDied","Data":"0e0cf6be47b357d3f80da917af1dc9b9fab0fd611e94ef3f88daf370c599f61b"} Oct 30 21:30:03.623404 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.623385 4926 scope.go:122] "RemoveContainer" containerID="0e0cf6be47b357d3f80da917af1dc9b9fab0fd611e94ef3f88daf370c599f61b" Oct 30 21:30:03.630760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.630740 4926 generic.go:356] "Generic (PLEG): container finished" podID="6d846cc6-bbdd-456a-a049-e9d99773172d" containerID="2d94a5e0e43a04fd006499ea0987d76ff99c8301c44103e16e7c580ae8fa3626" exitCode=60 Oct 30 21:30:03.630845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.630796 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4" event={"ID":"6d846cc6-bbdd-456a-a049-e9d99773172d","Type":"ContainerDied","Data":"2d94a5e0e43a04fd006499ea0987d76ff99c8301c44103e16e7c580ae8fa3626"} Oct 30 21:30:03.631233 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.631096 4926 scope.go:122] "RemoveContainer" containerID="2d94a5e0e43a04fd006499ea0987d76ff99c8301c44103e16e7c580ae8fa3626" Oct 30 21:30:03.633383 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.633355 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" event={"ID":"63543c7d-5e17-4248-82f2-b5b947e7aa4c","Type":"ContainerStarted","Data":"3221a7abcf1283862740bbafe2e88cc2c081dda6bd69a6ac3b031d745a5d2bdf"} Oct 30 21:30:03.636017 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.635995 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6" event={"ID":"810d266f-62a8-4b0f-a146-89ed2e41df99","Type":"ContainerStarted","Data":"91cbc2c0b77de7139c68c4db6d2886fe31e9fb1e6ef06e515ee934a6d9de1f18"} Oct 30 21:30:03.637660 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.637634 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9" event={"ID":"bde714d2-046b-443a-a5e6-f87c729e5ab0","Type":"ContainerStarted","Data":"735b3e3cce6c5b911bd96c00a5e0574d8103946b440a9d2dbb246945941a205f"} Oct 30 21:30:03.639231 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.639210 4926 generic.go:356] "Generic (PLEG): container finished" podID="731eb7a0-c9e9-41f9-b245-0fe7a00be673" containerID="5906c65d92616f8ac519864d6fa5b03a31f202bdbf89b028fa3b006cf0f16816" exitCode=60 Oct 30 21:30:03.639330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.639289 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8" event={"ID":"731eb7a0-c9e9-41f9-b245-0fe7a00be673","Type":"ContainerDied","Data":"5906c65d92616f8ac519864d6fa5b03a31f202bdbf89b028fa3b006cf0f16816"} Oct 30 21:30:03.639489 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.639470 4926 scope.go:122] "RemoveContainer" containerID="5906c65d92616f8ac519864d6fa5b03a31f202bdbf89b028fa3b006cf0f16816" Oct 30 21:30:03.640997 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.640972 4926 generic.go:356] "Generic (PLEG): container finished" podID="d922621f-a5d6-4854-8f0e-ba0ee21ade8e" containerID="a145722a0015db2ca9984859fc55da22f72328621238fb634a36dd2fd03b8c77" exitCode=60 Oct 30 21:30:03.641113 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.641085 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg" event={"ID":"d922621f-a5d6-4854-8f0e-ba0ee21ade8e","Type":"ContainerDied","Data":"a145722a0015db2ca9984859fc55da22f72328621238fb634a36dd2fd03b8c77"} Oct 30 21:30:03.641258 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.641242 4926 scope.go:122] "RemoveContainer" containerID="a145722a0015db2ca9984859fc55da22f72328621238fb634a36dd2fd03b8c77" Oct 30 21:30:03.642855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.642829 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq" event={"ID":"fce0f54b-5f27-4637-8b5c-d6f2298873bb","Type":"ContainerStarted","Data":"ae09596c42773baf21221129a3218dbdce0ae455eb7ad1dcdffdae48d00d0473"} Oct 30 21:30:03.644545 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.644525 4926 generic.go:356] "Generic (PLEG): container finished" podID="495acf58-df75-4721-aaeb-fa3dcb9a3cd9" containerID="7a4dc82a11f21b7109031a380be79af07d50c1f1616821d6ed9020473335ad3f" exitCode=60 Oct 30 21:30:03.644645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.644558 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" event={"ID":"495acf58-df75-4721-aaeb-fa3dcb9a3cd9","Type":"ContainerDied","Data":"7a4dc82a11f21b7109031a380be79af07d50c1f1616821d6ed9020473335ad3f"} Oct 30 21:30:03.644807 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.644789 4926 scope.go:122] "RemoveContainer" containerID="7a4dc82a11f21b7109031a380be79af07d50c1f1616821d6ed9020473335ad3f" Oct 30 21:30:03.647347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.647309 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" event={"ID":"f5d36899-edf3-4bb0-a13b-39c36bf0ed68","Type":"ContainerStarted","Data":"6a8c156c20c7b62e89eba360e41f1f88564fc6f5c1a5110b100cb593583a3f34"} Oct 30 21:30:03.649440 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.649422 4926 generic.go:356] "Generic (PLEG): container finished" podID="61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" containerID="fc2716b6916caa2faaea4f9e113d0f21dfd630c4d58ffdf63aac04fd2b32b929" exitCode=0 Oct 30 21:30:03.649541 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.649495 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" event={"ID":"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7","Type":"ContainerDied","Data":"fc2716b6916caa2faaea4f9e113d0f21dfd630c4d58ffdf63aac04fd2b32b929"} Oct 30 21:30:03.649921 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.649901 4926 scope.go:122] "RemoveContainer" containerID="fc2716b6916caa2faaea4f9e113d0f21dfd630c4d58ffdf63aac04fd2b32b929" Oct 30 21:30:03.651709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.651685 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" event={"ID":"0fe4049a-063d-4990-8524-88c95c2d8862","Type":"ContainerStarted","Data":"952074036b6de27afd99d8f66796206439351dbd4e41a7138de3a5f6960069ac"} Oct 30 21:30:03.653755 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.653723 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4" event={"ID":"f17fd537-658d-4c4f-9aeb-a763d184c364","Type":"ContainerStarted","Data":"63fcade6e209388abf46882c4e25e470c3fea61614e5cd95e3e13dae96d34fd4"} Oct 30 21:30:03.655954 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.655934 4926 generic.go:356] "Generic (PLEG): container finished" podID="d55b8ddb-6b3e-4f51-8281-f3209c63d949" containerID="25bfe40aca640e34ed8078248d04bcdfd28dcf27306b0b1c283f9ec234573f1a" exitCode=60 Oct 30 21:30:03.656122 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.656013 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc" event={"ID":"d55b8ddb-6b3e-4f51-8281-f3209c63d949","Type":"ContainerDied","Data":"25bfe40aca640e34ed8078248d04bcdfd28dcf27306b0b1c283f9ec234573f1a"} Oct 30 21:30:03.656309 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.656285 4926 scope.go:122] "RemoveContainer" containerID="25bfe40aca640e34ed8078248d04bcdfd28dcf27306b0b1c283f9ec234573f1a" Oct 30 21:30:03.657362 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.657336 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:30:03.657443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.657402 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:30:03.657443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.657435 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:30:03.799021 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.797317 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6" podStartSLOduration=3.797294713 podStartE2EDuration="3.797294713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.793117239 +0000 UTC m=+1790.914190980" watchObservedRunningTime="2026-10-30 21:30:03.797294713 +0000 UTC m=+1790.918368456" Oct 30 21:30:03.803164 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.803110 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4" podStartSLOduration=3.803091978 podStartE2EDuration="3.803091978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.750404808 +0000 UTC m=+1790.871478557" watchObservedRunningTime="2026-10-30 21:30:03.803091978 +0000 UTC m=+1790.924165725" Oct 30 21:30:03.837730 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.837686 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" podStartSLOduration=3.837660068 podStartE2EDuration="3.837660068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.834084615 +0000 UTC m=+1790.955158358" watchObservedRunningTime="2026-10-30 21:30:03.837660068 +0000 UTC m=+1790.958733792" Oct 30 21:30:03.957676 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.957624 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" podStartSLOduration=3.957608692 podStartE2EDuration="3.957608692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.952348866 +0000 UTC m=+1791.073422611" watchObservedRunningTime="2026-10-30 21:30:03.957608692 +0000 UTC m=+1791.078682438" Oct 30 21:30:03.994387 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:03.994320 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq" podStartSLOduration=3.9943007 podStartE2EDuration="3.9943007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:03.990400915 +0000 UTC m=+1791.111474662" watchObservedRunningTime="2026-10-30 21:30:03.9943007 +0000 UTC m=+1791.115374447" Oct 30 21:30:04.215080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.214744 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9" podStartSLOduration=4.214726443 podStartE2EDuration="4.214726443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:04.213689607 +0000 UTC m=+1791.334763353" watchObservedRunningTime="2026-10-30 21:30:04.214726443 +0000 UTC m=+1791.335800192" Oct 30 21:30:04.338339 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.338261 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" podStartSLOduration=4.338239467 podStartE2EDuration="4.338239467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:04.32247028 +0000 UTC m=+1791.443544027" watchObservedRunningTime="2026-10-30 21:30:04.338239467 +0000 UTC m=+1791.459313214" Oct 30 21:30:04.681910 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.679769 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg" event={"ID":"d922621f-a5d6-4854-8f0e-ba0ee21ade8e","Type":"ContainerStarted","Data":"c38e6d12b016304b0cf980f598a23f75e6816037432c95cf79a87c17f7ab10d5"} Oct 30 21:30:04.686815 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.685322 4926 generic.go:356] "Generic (PLEG): container finished" podID="fce0f54b-5f27-4637-8b5c-d6f2298873bb" containerID="ae09596c42773baf21221129a3218dbdce0ae455eb7ad1dcdffdae48d00d0473" exitCode=60 Oct 30 21:30:04.686815 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.685447 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq" event={"ID":"fce0f54b-5f27-4637-8b5c-d6f2298873bb","Type":"ContainerDied","Data":"ae09596c42773baf21221129a3218dbdce0ae455eb7ad1dcdffdae48d00d0473"} Oct 30 21:30:04.686815 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.685885 4926 scope.go:122] "RemoveContainer" containerID="ae09596c42773baf21221129a3218dbdce0ae455eb7ad1dcdffdae48d00d0473" Oct 30 21:30:04.700348 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.700319 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" event={"ID":"495acf58-df75-4721-aaeb-fa3dcb9a3cd9","Type":"ContainerStarted","Data":"c16157b7b56e36ac6c186eb4404beb0736686408a3c87fba717fc2c65e6bf0ae"} Oct 30 21:30:04.703268 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.703223 4926 generic.go:356] "Generic (PLEG): container finished" podID="f5d36899-edf3-4bb0-a13b-39c36bf0ed68" containerID="6a8c156c20c7b62e89eba360e41f1f88564fc6f5c1a5110b100cb593583a3f34" exitCode=60 Oct 30 21:30:04.703414 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.703398 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" event={"ID":"f5d36899-edf3-4bb0-a13b-39c36bf0ed68","Type":"ContainerDied","Data":"6a8c156c20c7b62e89eba360e41f1f88564fc6f5c1a5110b100cb593583a3f34"} Oct 30 21:30:04.704603 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.704404 4926 scope.go:122] "RemoveContainer" containerID="6a8c156c20c7b62e89eba360e41f1f88564fc6f5c1a5110b100cb593583a3f34" Oct 30 21:30:04.706879 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.706856 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" event={"ID":"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7","Type":"ContainerStarted","Data":"4a8984badf27ef286f33ed41fe398f5a939b4a8c838e0da9fcf4da9d9e2e4ed2"} Oct 30 21:30:04.712180 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.712154 4926 generic.go:356] "Generic (PLEG): container finished" podID="0fe4049a-063d-4990-8524-88c95c2d8862" containerID="952074036b6de27afd99d8f66796206439351dbd4e41a7138de3a5f6960069ac" exitCode=60 Oct 30 21:30:04.712283 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.712189 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" event={"ID":"0fe4049a-063d-4990-8524-88c95c2d8862","Type":"ContainerDied","Data":"952074036b6de27afd99d8f66796206439351dbd4e41a7138de3a5f6960069ac"} Oct 30 21:30:04.717362 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.713867 4926 scope.go:122] "RemoveContainer" containerID="952074036b6de27afd99d8f66796206439351dbd4e41a7138de3a5f6960069ac" Oct 30 21:30:04.718847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.718825 4926 generic.go:356] "Generic (PLEG): container finished" podID="f17fd537-658d-4c4f-9aeb-a763d184c364" containerID="63fcade6e209388abf46882c4e25e470c3fea61614e5cd95e3e13dae96d34fd4" exitCode=60 Oct 30 21:30:04.718960 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.718947 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4" event={"ID":"f17fd537-658d-4c4f-9aeb-a763d184c364","Type":"ContainerDied","Data":"63fcade6e209388abf46882c4e25e470c3fea61614e5cd95e3e13dae96d34fd4"} Oct 30 21:30:04.719691 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.719677 4926 scope.go:122] "RemoveContainer" containerID="63fcade6e209388abf46882c4e25e470c3fea61614e5cd95e3e13dae96d34fd4" Oct 30 21:30:04.721197 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.721181 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj" event={"ID":"c461be4b-e409-48fe-bcaf-6ba8e18ab312","Type":"ContainerStarted","Data":"66e79fcc45bec382b8615d88f30bac70f3efbdfa809a0d676a7df29c74ef77f1"} Oct 30 21:30:04.726476 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.725181 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9" event={"ID":"fd7f7c91-0516-476c-afc8-94130b2e3439","Type":"ContainerStarted","Data":"8cfce7c4425f57965a31ee041efb4a10aa13205af3c55c3652489affd3a4ed64"} Oct 30 21:30:04.727342 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.727295 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg" podStartSLOduration=4.727279029 podStartE2EDuration="4.727279029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:04.724999972 +0000 UTC m=+1791.846073718" watchObservedRunningTime="2026-10-30 21:30:04.727279029 +0000 UTC m=+1791.848352776" Oct 30 21:30:04.729847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.729811 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4" event={"ID":"6d846cc6-bbdd-456a-a049-e9d99773172d","Type":"ContainerStarted","Data":"5151329cc5da7e64d088252663ae0bf1a64daef1a827bf605b2531c0cc504630"} Oct 30 21:30:04.734230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.734196 4926 generic.go:356] "Generic (PLEG): container finished" podID="63543c7d-5e17-4248-82f2-b5b947e7aa4c" containerID="3221a7abcf1283862740bbafe2e88cc2c081dda6bd69a6ac3b031d745a5d2bdf" exitCode=60 Oct 30 21:30:04.734336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.734304 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" event={"ID":"63543c7d-5e17-4248-82f2-b5b947e7aa4c","Type":"ContainerDied","Data":"3221a7abcf1283862740bbafe2e88cc2c081dda6bd69a6ac3b031d745a5d2bdf"} Oct 30 21:30:04.734666 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.734647 4926 scope.go:122] "RemoveContainer" containerID="3221a7abcf1283862740bbafe2e88cc2c081dda6bd69a6ac3b031d745a5d2bdf" Oct 30 21:30:04.744347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.744323 4926 generic.go:356] "Generic (PLEG): container finished" podID="810d266f-62a8-4b0f-a146-89ed2e41df99" containerID="91cbc2c0b77de7139c68c4db6d2886fe31e9fb1e6ef06e515ee934a6d9de1f18" exitCode=60 Oct 30 21:30:04.744446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.744351 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6" event={"ID":"810d266f-62a8-4b0f-a146-89ed2e41df99","Type":"ContainerDied","Data":"91cbc2c0b77de7139c68c4db6d2886fe31e9fb1e6ef06e515ee934a6d9de1f18"} Oct 30 21:30:04.744837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.744815 4926 scope.go:122] "RemoveContainer" containerID="91cbc2c0b77de7139c68c4db6d2886fe31e9fb1e6ef06e515ee934a6d9de1f18" Oct 30 21:30:04.749240 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.749214 4926 generic.go:356] "Generic (PLEG): container finished" podID="bde714d2-046b-443a-a5e6-f87c729e5ab0" containerID="735b3e3cce6c5b911bd96c00a5e0574d8103946b440a9d2dbb246945941a205f" exitCode=60 Oct 30 21:30:04.749330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.749304 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9" event={"ID":"bde714d2-046b-443a-a5e6-f87c729e5ab0","Type":"ContainerDied","Data":"735b3e3cce6c5b911bd96c00a5e0574d8103946b440a9d2dbb246945941a205f"} Oct 30 21:30:04.750343 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.750315 4926 scope.go:122] "RemoveContainer" containerID="735b3e3cce6c5b911bd96c00a5e0574d8103946b440a9d2dbb246945941a205f" Oct 30 21:30:04.754358 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.754338 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8" event={"ID":"731eb7a0-c9e9-41f9-b245-0fe7a00be673","Type":"ContainerStarted","Data":"16529e09a3270761076d03523322758f43f467eeee987453ae987fca8d3657d9"} Oct 30 21:30:04.878953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:04.876481 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:30:05.121303 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.121255 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9" podStartSLOduration=5.121241379 podStartE2EDuration="5.121241379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:05.118215193 +0000 UTC m=+1792.239288940" watchObservedRunningTime="2026-10-30 21:30:05.121241379 +0000 UTC m=+1792.242315124" Oct 30 21:30:05.212131 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.212097 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:30:05.811058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.795863 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9" event={"ID":"bde714d2-046b-443a-a5e6-f87c729e5ab0","Type":"ContainerStarted","Data":"3c091fd1ecd2c8ebba43baf1f7776abd134bad43211b1ead4d6b787b329e52ba"} Oct 30 21:30:05.811058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.801559 4926 generic.go:356] "Generic (PLEG): container finished" podID="731eb7a0-c9e9-41f9-b245-0fe7a00be673" containerID="16529e09a3270761076d03523322758f43f467eeee987453ae987fca8d3657d9" exitCode=60 Oct 30 21:30:05.811058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.801676 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8" event={"ID":"731eb7a0-c9e9-41f9-b245-0fe7a00be673","Type":"ContainerDied","Data":"16529e09a3270761076d03523322758f43f467eeee987453ae987fca8d3657d9"} Oct 30 21:30:05.811058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.801729 4926 scope.go:122] "RemoveContainer" containerID="5906c65d92616f8ac519864d6fa5b03a31f202bdbf89b028fa3b006cf0f16816" Oct 30 21:30:05.811058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.806968 4926 generic.go:356] "Generic (PLEG): container finished" podID="d922621f-a5d6-4854-8f0e-ba0ee21ade8e" containerID="c38e6d12b016304b0cf980f598a23f75e6816037432c95cf79a87c17f7ab10d5" exitCode=60 Oct 30 21:30:05.811058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.807113 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg" event={"ID":"d922621f-a5d6-4854-8f0e-ba0ee21ade8e","Type":"ContainerDied","Data":"c38e6d12b016304b0cf980f598a23f75e6816037432c95cf79a87c17f7ab10d5"} Oct 30 21:30:05.811058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.810838 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq" event={"ID":"fce0f54b-5f27-4637-8b5c-d6f2298873bb","Type":"ContainerStarted","Data":"22bc22dac5ddf8a3084da6a7755feaeb397fd1cb4b8080eb8aa40e1753a94c3d"} Oct 30 21:30:05.813373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.813348 4926 generic.go:356] "Generic (PLEG): container finished" podID="495acf58-df75-4721-aaeb-fa3dcb9a3cd9" containerID="c16157b7b56e36ac6c186eb4404beb0736686408a3c87fba717fc2c65e6bf0ae" exitCode=60 Oct 30 21:30:05.813615 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.813551 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" event={"ID":"495acf58-df75-4721-aaeb-fa3dcb9a3cd9","Type":"ContainerDied","Data":"c16157b7b56e36ac6c186eb4404beb0736686408a3c87fba717fc2c65e6bf0ae"} Oct 30 21:30:05.816191 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.816169 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" event={"ID":"f5d36899-edf3-4bb0-a13b-39c36bf0ed68","Type":"ContainerStarted","Data":"6147dde711b6adf97bc900cb4178581e1f20ee2e5be7f8ca8a2375d1d923d68c"} Oct 30 21:30:05.819045 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.819006 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" event={"ID":"0fe4049a-063d-4990-8524-88c95c2d8862","Type":"ContainerStarted","Data":"d9bb99512c4d52c3ac2c4895353bda862dafa2d376f89298f0b5f0171b79ada4"} Oct 30 21:30:05.821960 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.821939 4926 scope.go:122] "RemoveContainer" containerID="16529e09a3270761076d03523322758f43f467eeee987453ae987fca8d3657d9" Oct 30 21:30:05.822500 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:05.822475 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8_odh-ai-gateway-infra(731eb7a0-c9e9-41f9-b245-0fe7a00be673)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8" podUID="731eb7a0-c9e9-41f9-b245-0fe7a00be673" Oct 30 21:30:05.823378 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.823359 4926 scope.go:122] "RemoveContainer" containerID="c38e6d12b016304b0cf980f598a23f75e6816037432c95cf79a87c17f7ab10d5" Oct 30 21:30:05.823793 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:05.823770 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg_odh-ai-gateway-infra(d922621f-a5d6-4854-8f0e-ba0ee21ade8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg" podUID="d922621f-a5d6-4854-8f0e-ba0ee21ade8e" Oct 30 21:30:05.823933 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.823919 4926 scope.go:122] "RemoveContainer" containerID="c16157b7b56e36ac6c186eb4404beb0736686408a3c87fba717fc2c65e6bf0ae" Oct 30 21:30:05.824373 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:05.824352 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv_odh-ai-gateway-infra(495acf58-df75-4721-aaeb-fa3dcb9a3cd9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" podUID="495acf58-df75-4721-aaeb-fa3dcb9a3cd9" Oct 30 21:30:05.824910 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.824889 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4" event={"ID":"f17fd537-658d-4c4f-9aeb-a763d184c364","Type":"ContainerStarted","Data":"7ae5b8b41f62bd2dc3fc4b90234d7e820a3c4b140230b012ae4b71f583e06150"} Oct 30 21:30:05.827558 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.827537 4926 generic.go:356] "Generic (PLEG): container finished" podID="d55b8ddb-6b3e-4f51-8281-f3209c63d949" containerID="e156e111879c18643c59b38af35a0dc21f5f958b7947d0534a78c93014a5531c" exitCode=60 Oct 30 21:30:05.827661 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.827644 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc" event={"ID":"d55b8ddb-6b3e-4f51-8281-f3209c63d949","Type":"ContainerDied","Data":"e156e111879c18643c59b38af35a0dc21f5f958b7947d0534a78c93014a5531c"} Oct 30 21:30:05.828172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.828138 4926 scope.go:122] "RemoveContainer" containerID="e156e111879c18643c59b38af35a0dc21f5f958b7947d0534a78c93014a5531c" Oct 30 21:30:05.828560 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:05.828536 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-lmlbc_odh-ai-gateway-infra(d55b8ddb-6b3e-4f51-8281-f3209c63d949)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc" podUID="d55b8ddb-6b3e-4f51-8281-f3209c63d949" Oct 30 21:30:05.829963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.829939 4926 generic.go:356] "Generic (PLEG): container finished" podID="c461be4b-e409-48fe-bcaf-6ba8e18ab312" containerID="66e79fcc45bec382b8615d88f30bac70f3efbdfa809a0d676a7df29c74ef77f1" exitCode=60 Oct 30 21:30:05.830088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.830071 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj" event={"ID":"c461be4b-e409-48fe-bcaf-6ba8e18ab312","Type":"ContainerDied","Data":"66e79fcc45bec382b8615d88f30bac70f3efbdfa809a0d676a7df29c74ef77f1"} Oct 30 21:30:05.830217 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.830192 4926 scope.go:122] "RemoveContainer" containerID="66e79fcc45bec382b8615d88f30bac70f3efbdfa809a0d676a7df29c74ef77f1" Oct 30 21:30:05.830397 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:05.830376 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj_odh-ai-gateway-infra(c461be4b-e409-48fe-bcaf-6ba8e18ab312)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj" podUID="c461be4b-e409-48fe-bcaf-6ba8e18ab312" Oct 30 21:30:05.833428 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.833408 4926 generic.go:356] "Generic (PLEG): container finished" podID="fd7f7c91-0516-476c-afc8-94130b2e3439" containerID="8cfce7c4425f57965a31ee041efb4a10aa13205af3c55c3652489affd3a4ed64" exitCode=60 Oct 30 21:30:05.833535 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.833483 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9" event={"ID":"fd7f7c91-0516-476c-afc8-94130b2e3439","Type":"ContainerDied","Data":"8cfce7c4425f57965a31ee041efb4a10aa13205af3c55c3652489affd3a4ed64"} Oct 30 21:30:05.833756 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.833740 4926 scope.go:122] "RemoveContainer" containerID="8cfce7c4425f57965a31ee041efb4a10aa13205af3c55c3652489affd3a4ed64" Oct 30 21:30:05.833956 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:05.833938 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9_odh-ai-gateway-infra(fd7f7c91-0516-476c-afc8-94130b2e3439)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9" podUID="fd7f7c91-0516-476c-afc8-94130b2e3439" Oct 30 21:30:05.835626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.835607 4926 generic.go:356] "Generic (PLEG): container finished" podID="6d846cc6-bbdd-456a-a049-e9d99773172d" containerID="5151329cc5da7e64d088252663ae0bf1a64daef1a827bf605b2531c0cc504630" exitCode=60 Oct 30 21:30:05.835838 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.835821 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4" event={"ID":"6d846cc6-bbdd-456a-a049-e9d99773172d","Type":"ContainerDied","Data":"5151329cc5da7e64d088252663ae0bf1a64daef1a827bf605b2531c0cc504630"} Oct 30 21:30:05.836267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.836252 4926 scope.go:122] "RemoveContainer" containerID="5151329cc5da7e64d088252663ae0bf1a64daef1a827bf605b2531c0cc504630" Oct 30 21:30:05.836754 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.836733 4926 scope.go:122] "RemoveContainer" containerID="a145722a0015db2ca9984859fc55da22f72328621238fb634a36dd2fd03b8c77" Oct 30 21:30:05.836886 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:05.836864 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4_odh-ai-gateway-infra(6d846cc6-bbdd-456a-a049-e9d99773172d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4" podUID="6d846cc6-bbdd-456a-a049-e9d99773172d" Oct 30 21:30:05.839185 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.839164 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" event={"ID":"63543c7d-5e17-4248-82f2-b5b947e7aa4c","Type":"ContainerStarted","Data":"9275fab04d8ed86827ce84b220682141d3bbe0315748b9c8d8318d2dbd4e2b0c"} Oct 30 21:30:05.843149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.843124 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6" event={"ID":"810d266f-62a8-4b0f-a146-89ed2e41df99","Type":"ContainerStarted","Data":"f6019dd0da685ea4b454854fc35d7995182043f1a0afabcdc9d125bb28330750"} Oct 30 21:30:05.855975 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.855953 4926 scope.go:122] "RemoveContainer" containerID="7a4dc82a11f21b7109031a380be79af07d50c1f1616821d6ed9020473335ad3f" Oct 30 21:30:05.881794 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.881754 4926 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 21:30:05.882382 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.882351 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" podUID="61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 21:30:05.935701 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.935567 4926 scope.go:122] "RemoveContainer" containerID="25bfe40aca640e34ed8078248d04bcdfd28dcf27306b0b1c283f9ec234573f1a" Oct 30 21:30:05.953508 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:05.953487 4926 scope.go:122] "RemoveContainer" containerID="4e9db721d59e6f0af872ce51a59d73d93d9b46511b397318b4e591a6734363ad" Oct 30 21:30:06.018978 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.018952 4926 scope.go:122] "RemoveContainer" containerID="0e0cf6be47b357d3f80da917af1dc9b9fab0fd611e94ef3f88daf370c599f61b" Oct 30 21:30:06.035727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.035656 4926 scope.go:122] "RemoveContainer" containerID="2d94a5e0e43a04fd006499ea0987d76ff99c8301c44103e16e7c580ae8fa3626" Oct 30 21:30:06.858019 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.857976 4926 scope.go:122] "RemoveContainer" containerID="c16157b7b56e36ac6c186eb4404beb0736686408a3c87fba717fc2c65e6bf0ae" Oct 30 21:30:06.858464 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:06.858438 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv_odh-ai-gateway-infra(495acf58-df75-4721-aaeb-fa3dcb9a3cd9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" podUID="495acf58-df75-4721-aaeb-fa3dcb9a3cd9" Oct 30 21:30:06.859893 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.859865 4926 generic.go:356] "Generic (PLEG): container finished" podID="f5d36899-edf3-4bb0-a13b-39c36bf0ed68" containerID="6147dde711b6adf97bc900cb4178581e1f20ee2e5be7f8ca8a2375d1d923d68c" exitCode=60 Oct 30 21:30:06.860064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.859935 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" event={"ID":"f5d36899-edf3-4bb0-a13b-39c36bf0ed68","Type":"ContainerDied","Data":"6147dde711b6adf97bc900cb4178581e1f20ee2e5be7f8ca8a2375d1d923d68c"} Oct 30 21:30:06.860064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.860018 4926 scope.go:122] "RemoveContainer" containerID="6a8c156c20c7b62e89eba360e41f1f88564fc6f5c1a5110b100cb593583a3f34" Oct 30 21:30:06.860368 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.860348 4926 scope.go:122] "RemoveContainer" containerID="6147dde711b6adf97bc900cb4178581e1f20ee2e5be7f8ca8a2375d1d923d68c" Oct 30 21:30:06.860844 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:06.860815 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r_odh-ai-gateway-infra(f5d36899-edf3-4bb0-a13b-39c36bf0ed68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" podUID="f5d36899-edf3-4bb0-a13b-39c36bf0ed68" Oct 30 21:30:06.862239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.862216 4926 generic.go:356] "Generic (PLEG): container finished" podID="0fe4049a-063d-4990-8524-88c95c2d8862" containerID="d9bb99512c4d52c3ac2c4895353bda862dafa2d376f89298f0b5f0171b79ada4" exitCode=60 Oct 30 21:30:06.862322 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.862297 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" event={"ID":"0fe4049a-063d-4990-8524-88c95c2d8862","Type":"ContainerDied","Data":"d9bb99512c4d52c3ac2c4895353bda862dafa2d376f89298f0b5f0171b79ada4"} Oct 30 21:30:06.862587 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.862566 4926 scope.go:122] "RemoveContainer" containerID="d9bb99512c4d52c3ac2c4895353bda862dafa2d376f89298f0b5f0171b79ada4" Oct 30 21:30:06.862958 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:06.862935 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28_odh-ai-gateway-infra(0fe4049a-063d-4990-8524-88c95c2d8862)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" podUID="0fe4049a-063d-4990-8524-88c95c2d8862" Oct 30 21:30:06.864360 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.864340 4926 generic.go:356] "Generic (PLEG): container finished" podID="f17fd537-658d-4c4f-9aeb-a763d184c364" containerID="7ae5b8b41f62bd2dc3fc4b90234d7e820a3c4b140230b012ae4b71f583e06150" exitCode=60 Oct 30 21:30:06.864450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.864371 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4" event={"ID":"f17fd537-658d-4c4f-9aeb-a763d184c364","Type":"ContainerDied","Data":"7ae5b8b41f62bd2dc3fc4b90234d7e820a3c4b140230b012ae4b71f583e06150"} Oct 30 21:30:06.864660 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.864643 4926 scope.go:122] "RemoveContainer" containerID="7ae5b8b41f62bd2dc3fc4b90234d7e820a3c4b140230b012ae4b71f583e06150" Oct 30 21:30:06.864984 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:06.864962 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4_odh-ai-gateway-infra(f17fd537-658d-4c4f-9aeb-a763d184c364)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4" podUID="f17fd537-658d-4c4f-9aeb-a763d184c364" Oct 30 21:30:06.879051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.866322 4926 scope.go:122] "RemoveContainer" containerID="e156e111879c18643c59b38af35a0dc21f5f958b7947d0534a78c93014a5531c" Oct 30 21:30:06.879051 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:06.866614 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-lmlbc_odh-ai-gateway-infra(d55b8ddb-6b3e-4f51-8281-f3209c63d949)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc" podUID="d55b8ddb-6b3e-4f51-8281-f3209c63d949" Oct 30 21:30:06.896053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.884166 4926 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 21:30:06.896053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.884230 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" podUID="61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 21:30:06.905699 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.905667 4926 scope.go:122] "RemoveContainer" containerID="66e79fcc45bec382b8615d88f30bac70f3efbdfa809a0d676a7df29c74ef77f1" Oct 30 21:30:06.906399 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:06.906368 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj_odh-ai-gateway-infra(c461be4b-e409-48fe-bcaf-6ba8e18ab312)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj" podUID="c461be4b-e409-48fe-bcaf-6ba8e18ab312" Oct 30 21:30:06.911871 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.911848 4926 scope.go:122] "RemoveContainer" containerID="8cfce7c4425f57965a31ee041efb4a10aa13205af3c55c3652489affd3a4ed64" Oct 30 21:30:06.912413 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:06.912384 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9_odh-ai-gateway-infra(fd7f7c91-0516-476c-afc8-94130b2e3439)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9" podUID="fd7f7c91-0516-476c-afc8-94130b2e3439" Oct 30 21:30:06.914131 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.914109 4926 scope.go:122] "RemoveContainer" containerID="5151329cc5da7e64d088252663ae0bf1a64daef1a827bf605b2531c0cc504630" Oct 30 21:30:06.914395 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:06.914373 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4_odh-ai-gateway-infra(6d846cc6-bbdd-456a-a049-e9d99773172d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4" podUID="6d846cc6-bbdd-456a-a049-e9d99773172d" Oct 30 21:30:06.916860 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.916833 4926 generic.go:356] "Generic (PLEG): container finished" podID="63543c7d-5e17-4248-82f2-b5b947e7aa4c" containerID="9275fab04d8ed86827ce84b220682141d3bbe0315748b9c8d8318d2dbd4e2b0c" exitCode=60 Oct 30 21:30:06.916972 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.916947 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" event={"ID":"63543c7d-5e17-4248-82f2-b5b947e7aa4c","Type":"ContainerDied","Data":"9275fab04d8ed86827ce84b220682141d3bbe0315748b9c8d8318d2dbd4e2b0c"} Oct 30 21:30:06.917360 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.917338 4926 scope.go:122] "RemoveContainer" containerID="9275fab04d8ed86827ce84b220682141d3bbe0315748b9c8d8318d2dbd4e2b0c" Oct 30 21:30:06.917789 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:06.917763 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6_odh-ai-gateway-infra(63543c7d-5e17-4248-82f2-b5b947e7aa4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" podUID="63543c7d-5e17-4248-82f2-b5b947e7aa4c" Oct 30 21:30:06.920136 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.920114 4926 generic.go:356] "Generic (PLEG): container finished" podID="810d266f-62a8-4b0f-a146-89ed2e41df99" containerID="f6019dd0da685ea4b454854fc35d7995182043f1a0afabcdc9d125bb28330750" exitCode=60 Oct 30 21:30:06.920391 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.920265 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6" event={"ID":"810d266f-62a8-4b0f-a146-89ed2e41df99","Type":"ContainerDied","Data":"f6019dd0da685ea4b454854fc35d7995182043f1a0afabcdc9d125bb28330750"} Oct 30 21:30:06.920534 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.920473 4926 scope.go:122] "RemoveContainer" containerID="f6019dd0da685ea4b454854fc35d7995182043f1a0afabcdc9d125bb28330750" Oct 30 21:30:06.920966 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:06.920941 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6_odh-ai-gateway-infra(810d266f-62a8-4b0f-a146-89ed2e41df99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6" podUID="810d266f-62a8-4b0f-a146-89ed2e41df99" Oct 30 21:30:06.922791 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.922771 4926 generic.go:356] "Generic (PLEG): container finished" podID="bde714d2-046b-443a-a5e6-f87c729e5ab0" containerID="3c091fd1ecd2c8ebba43baf1f7776abd134bad43211b1ead4d6b787b329e52ba" exitCode=60 Oct 30 21:30:06.922970 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.922951 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9" event={"ID":"bde714d2-046b-443a-a5e6-f87c729e5ab0","Type":"ContainerDied","Data":"3c091fd1ecd2c8ebba43baf1f7776abd134bad43211b1ead4d6b787b329e52ba"} Oct 30 21:30:06.923110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.923092 4926 scope.go:122] "RemoveContainer" containerID="3c091fd1ecd2c8ebba43baf1f7776abd134bad43211b1ead4d6b787b329e52ba" Oct 30 21:30:06.923410 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:06.923385 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9_odh-ai-gateway-infra(bde714d2-046b-443a-a5e6-f87c729e5ab0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9" podUID="bde714d2-046b-443a-a5e6-f87c729e5ab0" Oct 30 21:30:06.928838 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.928795 4926 scope.go:122] "RemoveContainer" containerID="16529e09a3270761076d03523322758f43f467eeee987453ae987fca8d3657d9" Oct 30 21:30:06.929272 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:06.929244 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8_odh-ai-gateway-infra(731eb7a0-c9e9-41f9-b245-0fe7a00be673)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8" podUID="731eb7a0-c9e9-41f9-b245-0fe7a00be673" Oct 30 21:30:06.931876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.931855 4926 scope.go:122] "RemoveContainer" containerID="c38e6d12b016304b0cf980f598a23f75e6816037432c95cf79a87c17f7ab10d5" Oct 30 21:30:06.932136 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:06.932113 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg_odh-ai-gateway-infra(d922621f-a5d6-4854-8f0e-ba0ee21ade8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg" podUID="d922621f-a5d6-4854-8f0e-ba0ee21ade8e" Oct 30 21:30:06.934584 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.934563 4926 generic.go:356] "Generic (PLEG): container finished" podID="fce0f54b-5f27-4637-8b5c-d6f2298873bb" containerID="22bc22dac5ddf8a3084da6a7755feaeb397fd1cb4b8080eb8aa40e1753a94c3d" exitCode=60 Oct 30 21:30:06.934751 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.934627 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq" event={"ID":"fce0f54b-5f27-4637-8b5c-d6f2298873bb","Type":"ContainerDied","Data":"22bc22dac5ddf8a3084da6a7755feaeb397fd1cb4b8080eb8aa40e1753a94c3d"} Oct 30 21:30:06.934832 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.934815 4926 scope.go:122] "RemoveContainer" containerID="22bc22dac5ddf8a3084da6a7755feaeb397fd1cb4b8080eb8aa40e1753a94c3d" Oct 30 21:30:06.935266 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:06.935245 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq_odh-ai-gateway-infra(fce0f54b-5f27-4637-8b5c-d6f2298873bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq" podUID="fce0f54b-5f27-4637-8b5c-d6f2298873bb" Oct 30 21:30:06.935375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.935270 4926 scope.go:122] "RemoveContainer" containerID="952074036b6de27afd99d8f66796206439351dbd4e41a7138de3a5f6960069ac" Oct 30 21:30:06.950538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.950519 4926 scope.go:122] "RemoveContainer" containerID="63fcade6e209388abf46882c4e25e470c3fea61614e5cd95e3e13dae96d34fd4" Oct 30 21:30:06.971174 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:06.971148 4926 scope.go:122] "RemoveContainer" containerID="3221a7abcf1283862740bbafe2e88cc2c081dda6bd69a6ac3b031d745a5d2bdf" Oct 30 21:30:07.023520 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:07.023495 4926 scope.go:122] "RemoveContainer" containerID="91cbc2c0b77de7139c68c4db6d2886fe31e9fb1e6ef06e515ee934a6d9de1f18" Oct 30 21:30:07.038816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:07.038794 4926 scope.go:122] "RemoveContainer" containerID="735b3e3cce6c5b911bd96c00a5e0574d8103946b440a9d2dbb246945941a205f" Oct 30 21:30:07.055757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:07.055734 4926 scope.go:122] "RemoveContainer" containerID="ae09596c42773baf21221129a3218dbdce0ae455eb7ad1dcdffdae48d00d0473" Oct 30 21:30:07.525305 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:07.524679 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:30:07.900260 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:07.900174 4926 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 21:30:07.900727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:07.900593 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" podUID="61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 21:30:07.946542 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:07.946515 4926 scope.go:122] "RemoveContainer" containerID="f6019dd0da685ea4b454854fc35d7995182043f1a0afabcdc9d125bb28330750" Oct 30 21:30:07.946893 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:07.946867 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6_odh-ai-gateway-infra(810d266f-62a8-4b0f-a146-89ed2e41df99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6" podUID="810d266f-62a8-4b0f-a146-89ed2e41df99" Oct 30 21:30:07.948023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:07.947998 4926 scope.go:122] "RemoveContainer" containerID="3c091fd1ecd2c8ebba43baf1f7776abd134bad43211b1ead4d6b787b329e52ba" Oct 30 21:30:07.948383 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:07.948360 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9_odh-ai-gateway-infra(bde714d2-046b-443a-a5e6-f87c729e5ab0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9" podUID="bde714d2-046b-443a-a5e6-f87c729e5ab0" Oct 30 21:30:07.949603 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:07.949578 4926 scope.go:122] "RemoveContainer" containerID="22bc22dac5ddf8a3084da6a7755feaeb397fd1cb4b8080eb8aa40e1753a94c3d" Oct 30 21:30:07.949859 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:07.949837 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq_odh-ai-gateway-infra(fce0f54b-5f27-4637-8b5c-d6f2298873bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq" podUID="fce0f54b-5f27-4637-8b5c-d6f2298873bb" Oct 30 21:30:07.951118 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:07.951098 4926 scope.go:122] "RemoveContainer" containerID="6147dde711b6adf97bc900cb4178581e1f20ee2e5be7f8ca8a2375d1d923d68c" Oct 30 21:30:07.951351 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:07.951331 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r_odh-ai-gateway-infra(f5d36899-edf3-4bb0-a13b-39c36bf0ed68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" podUID="f5d36899-edf3-4bb0-a13b-39c36bf0ed68" Oct 30 21:30:07.952818 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:07.952793 4926 scope.go:122] "RemoveContainer" containerID="d9bb99512c4d52c3ac2c4895353bda862dafa2d376f89298f0b5f0171b79ada4" Oct 30 21:30:07.953142 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:07.953120 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28_odh-ai-gateway-infra(0fe4049a-063d-4990-8524-88c95c2d8862)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" podUID="0fe4049a-063d-4990-8524-88c95c2d8862" Oct 30 21:30:07.955318 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:07.955297 4926 scope.go:122] "RemoveContainer" containerID="7ae5b8b41f62bd2dc3fc4b90234d7e820a3c4b140230b012ae4b71f583e06150" Oct 30 21:30:07.955602 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:07.955576 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4_odh-ai-gateway-infra(f17fd537-658d-4c4f-9aeb-a763d184c364)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4" podUID="f17fd537-658d-4c4f-9aeb-a763d184c364" Oct 30 21:30:07.957642 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:07.957622 4926 scope.go:122] "RemoveContainer" containerID="9275fab04d8ed86827ce84b220682141d3bbe0315748b9c8d8318d2dbd4e2b0c" Oct 30 21:30:07.957913 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:07.957892 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6_odh-ai-gateway-infra(63543c7d-5e17-4248-82f2-b5b947e7aa4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" podUID="63543c7d-5e17-4248-82f2-b5b947e7aa4c" Oct 30 21:30:08.884057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:08.880486 4926 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 21:30:08.884057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:08.880556 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" podUID="61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 21:30:09.884056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:09.873356 4926 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 21:30:09.884056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:09.873418 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" podUID="61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 21:30:10.334906 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:10.334864 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:30:10.349460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:10.349436 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:30:10.875062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:10.874326 4926 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Oct 30 21:30:10.875062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:10.874386 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" podUID="61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Oct 30 21:30:11.060607 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:11.060578 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:30:11.894051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:11.891331 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:30:11.894051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:11.893633 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:30:13.429991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:13.429954 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:30:13.433612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:13.433587 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:30:15.438399 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:15.438359 4926 scope.go:122] "RemoveContainer" containerID="6652831f2eafc1ec18c19be3f78d64ce09e1e9f584a45dd4dc13905e8c57bc2a" Oct 30 21:30:15.451208 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:15.451185 4926 scope.go:122] "RemoveContainer" containerID="7820a46d321e6dbb0d95db8bb327673d6600de4e4a12c7752452a9cf16bd2c31" Oct 30 21:30:15.461840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:15.461818 4926 scope.go:122] "RemoveContainer" containerID="30bf9242bd8515e99f83c989685a860ab754da1238ed88ed05f6c9c43d1d4994" Oct 30 21:30:16.009049 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:16.009002 4926 scope.go:122] "RemoveContainer" containerID="c38e6d12b016304b0cf980f598a23f75e6816037432c95cf79a87c17f7ab10d5" Oct 30 21:30:16.009250 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:16.009140 4926 scope.go:122] "RemoveContainer" containerID="66e79fcc45bec382b8615d88f30bac70f3efbdfa809a0d676a7df29c74ef77f1" Oct 30 21:30:16.009331 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:16.009296 4926 scope.go:122] "RemoveContainer" containerID="8cfce7c4425f57965a31ee041efb4a10aa13205af3c55c3652489affd3a4ed64" Oct 30 21:30:17.008978 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:17.008943 4926 scope.go:122] "RemoveContainer" containerID="16529e09a3270761076d03523322758f43f467eeee987453ae987fca8d3657d9" Oct 30 21:30:17.009498 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:17.009200 4926 scope.go:122] "RemoveContainer" containerID="5151329cc5da7e64d088252663ae0bf1a64daef1a827bf605b2531c0cc504630" Oct 30 21:30:17.009498 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:17.009376 4926 scope.go:122] "RemoveContainer" containerID="7ae5b8b41f62bd2dc3fc4b90234d7e820a3c4b140230b012ae4b71f583e06150" Oct 30 21:30:17.137685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:17.137657 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj" event={"ID":"c461be4b-e409-48fe-bcaf-6ba8e18ab312","Type":"ContainerStarted","Data":"9e82baae24716dafa3d2eb828c55b7c0e5ab8438a2344e505b5fc451d6700828"} Oct 30 21:30:17.139652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:17.139628 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9" event={"ID":"fd7f7c91-0516-476c-afc8-94130b2e3439","Type":"ContainerStarted","Data":"89b3873d7684979ca950b9a68ece1041fbb881ed25448baa02e3fa471ec32956"} Oct 30 21:30:17.141437 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:17.141410 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg" event={"ID":"d922621f-a5d6-4854-8f0e-ba0ee21ade8e","Type":"ContainerStarted","Data":"7423efedcd8a3b136fef7f6c4ca8304045012844ffe518be73bd28f98f7dccbb"} Oct 30 21:30:18.003054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.002272 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:30:18.017056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.014786 4926 scope.go:122] "RemoveContainer" containerID="3c091fd1ecd2c8ebba43baf1f7776abd134bad43211b1ead4d6b787b329e52ba" Oct 30 21:30:18.017056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.015861 4926 scope.go:122] "RemoveContainer" containerID="22bc22dac5ddf8a3084da6a7755feaeb397fd1cb4b8080eb8aa40e1753a94c3d" Oct 30 21:30:18.017619 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.017344 4926 scope.go:122] "RemoveContainer" containerID="f6019dd0da685ea4b454854fc35d7995182043f1a0afabcdc9d125bb28330750" Oct 30 21:30:18.242921 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.242882 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj"] Oct 30 21:30:18.248711 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.248662 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8" event={"ID":"731eb7a0-c9e9-41f9-b245-0fe7a00be673","Type":"ContainerStarted","Data":"4c501f3819b8567eae77a41737f0eae4c992296f651bbc1a0b813299bf1f4cf6"} Oct 30 21:30:18.252842 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.252812 4926 generic.go:356] "Generic (PLEG): container finished" podID="d922621f-a5d6-4854-8f0e-ba0ee21ade8e" containerID="7423efedcd8a3b136fef7f6c4ca8304045012844ffe518be73bd28f98f7dccbb" exitCode=60 Oct 30 21:30:18.253352 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.253282 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg" event={"ID":"d922621f-a5d6-4854-8f0e-ba0ee21ade8e","Type":"ContainerDied","Data":"7423efedcd8a3b136fef7f6c4ca8304045012844ffe518be73bd28f98f7dccbb"} Oct 30 21:30:18.253561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.253543 4926 scope.go:122] "RemoveContainer" containerID="c38e6d12b016304b0cf980f598a23f75e6816037432c95cf79a87c17f7ab10d5" Oct 30 21:30:18.261425 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.261402 4926 scope.go:122] "RemoveContainer" containerID="7423efedcd8a3b136fef7f6c4ca8304045012844ffe518be73bd28f98f7dccbb" Oct 30 21:30:18.262068 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:18.262023 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg_odh-ai-gateway-infra(d922621f-a5d6-4854-8f0e-ba0ee21ade8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg" podUID="d922621f-a5d6-4854-8f0e-ba0ee21ade8e" Oct 30 21:30:18.293277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.292014 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg"] Oct 30 21:30:18.315051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.308813 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4" event={"ID":"f17fd537-658d-4c4f-9aeb-a763d184c364","Type":"ContainerStarted","Data":"af90c2709ac2900d32a7435147c99776276fb4b4ae4d581be32fdad915b1a3d5"} Oct 30 21:30:18.326137 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.326108 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9"] Oct 30 21:30:18.333669 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.333632 4926 generic.go:356] "Generic (PLEG): container finished" podID="c461be4b-e409-48fe-bcaf-6ba8e18ab312" containerID="9e82baae24716dafa3d2eb828c55b7c0e5ab8438a2344e505b5fc451d6700828" exitCode=60 Oct 30 21:30:18.333904 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.333752 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj" event={"ID":"c461be4b-e409-48fe-bcaf-6ba8e18ab312","Type":"ContainerDied","Data":"9e82baae24716dafa3d2eb828c55b7c0e5ab8438a2344e505b5fc451d6700828"} Oct 30 21:30:18.340632 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.339102 4926 generic.go:356] "Generic (PLEG): container finished" podID="fd7f7c91-0516-476c-afc8-94130b2e3439" containerID="89b3873d7684979ca950b9a68ece1041fbb881ed25448baa02e3fa471ec32956" exitCode=60 Oct 30 21:30:18.340632 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.339203 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9" event={"ID":"fd7f7c91-0516-476c-afc8-94130b2e3439","Type":"ContainerDied","Data":"89b3873d7684979ca950b9a68ece1041fbb881ed25448baa02e3fa471ec32956"} Oct 30 21:30:18.346314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.345460 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4" event={"ID":"6d846cc6-bbdd-456a-a049-e9d99773172d","Type":"ContainerStarted","Data":"b49204426ca44e188614c4889ad6356be236fdafb3b71dcfbb7fd6027e36c404"} Oct 30 21:30:18.391066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.376698 4926 scope.go:122] "RemoveContainer" containerID="66e79fcc45bec382b8615d88f30bac70f3efbdfa809a0d676a7df29c74ef77f1" Oct 30 21:30:18.553732 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.553704 4926 scope.go:122] "RemoveContainer" containerID="8cfce7c4425f57965a31ee041efb4a10aa13205af3c55c3652489affd3a4ed64" Oct 30 21:30:18.691070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.686951 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9" Oct 30 21:30:18.797053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.792850 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7f7c91-0516-476c-afc8-94130b2e3439-kube-api-access-jwtsw\" (UniqueName: \"kubernetes.io/projected/fd7f7c91-0516-476c-afc8-94130b2e3439-kube-api-access-jwtsw\") pod \"fd7f7c91-0516-476c-afc8-94130b2e3439\" (UID: \"fd7f7c91-0516-476c-afc8-94130b2e3439\") " Oct 30 21:30:18.797053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.793071 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd7f7c91-0516-476c-afc8-94130b2e3439-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd7f7c91-0516-476c-afc8-94130b2e3439-maas-api-service-ca\") pod \"fd7f7c91-0516-476c-afc8-94130b2e3439\" (UID: \"fd7f7c91-0516-476c-afc8-94130b2e3439\") " Oct 30 21:30:18.854056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.798681 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd7f7c91-0516-476c-afc8-94130b2e3439-maas-api-service-ca" pod "fd7f7c91-0516-476c-afc8-94130b2e3439" (UID: "fd7f7c91-0516-476c-afc8-94130b2e3439"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:18.979312 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.883219 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7f7c91-0516-476c-afc8-94130b2e3439-kube-api-access-jwtsw" pod "fd7f7c91-0516-476c-afc8-94130b2e3439" (UID: "fd7f7c91-0516-476c-afc8-94130b2e3439"). InnerVolumeSpecName "kube-api-access-jwtsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:18.979312 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.897561 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd7f7c91-0516-476c-afc8-94130b2e3439-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:18.979312 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:18.897596 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwtsw\" (UniqueName: \"kubernetes.io/projected/fd7f7c91-0516-476c-afc8-94130b2e3439-kube-api-access-jwtsw\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:19.018633 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.013903 4926 scope.go:122] "RemoveContainer" containerID="e156e111879c18643c59b38af35a0dc21f5f958b7947d0534a78c93014a5531c" Oct 30 21:30:19.018633 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.015664 4926 scope.go:122] "RemoveContainer" containerID="6147dde711b6adf97bc900cb4178581e1f20ee2e5be7f8ca8a2375d1d923d68c" Oct 30 21:30:19.018633 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.016462 4926 scope.go:122] "RemoveContainer" containerID="c16157b7b56e36ac6c186eb4404beb0736686408a3c87fba717fc2c65e6bf0ae" Oct 30 21:30:19.018633 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.017956 4926 scope.go:122] "RemoveContainer" containerID="9275fab04d8ed86827ce84b220682141d3bbe0315748b9c8d8318d2dbd4e2b0c" Oct 30 21:30:19.141520 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.130438 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj" Oct 30 21:30:19.164106 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.145425 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78"] Oct 30 21:30:19.164106 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.145697 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" podUID="87424b1f-d6bc-42f6-ad13-b85cc826043f" containerName="istio-proxy" containerID="cri-o://6fab60554de72477679321feab0725fd0923c88c86b71079abd8285f67784890" gracePeriod=30 Oct 30 21:30:19.230533 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.230479 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c461be4b-e409-48fe-bcaf-6ba8e18ab312-maas-api-service-ca" pod "c461be4b-e409-48fe-bcaf-6ba8e18ab312" (UID: "c461be4b-e409-48fe-bcaf-6ba8e18ab312"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:19.231232 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.231196 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c461be4b-e409-48fe-bcaf-6ba8e18ab312-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c461be4b-e409-48fe-bcaf-6ba8e18ab312-maas-api-service-ca\") pod \"c461be4b-e409-48fe-bcaf-6ba8e18ab312\" (UID: \"c461be4b-e409-48fe-bcaf-6ba8e18ab312\") " Oct 30 21:30:19.245247 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.245216 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c461be4b-e409-48fe-bcaf-6ba8e18ab312-kube-api-access-2r2l6\" (UniqueName: \"kubernetes.io/projected/c461be4b-e409-48fe-bcaf-6ba8e18ab312-kube-api-access-2r2l6\") pod \"c461be4b-e409-48fe-bcaf-6ba8e18ab312\" (UID: \"c461be4b-e409-48fe-bcaf-6ba8e18ab312\") " Oct 30 21:30:19.248224 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.248194 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c461be4b-e409-48fe-bcaf-6ba8e18ab312-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:19.309757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.309718 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c461be4b-e409-48fe-bcaf-6ba8e18ab312-kube-api-access-2r2l6" pod "c461be4b-e409-48fe-bcaf-6ba8e18ab312" (UID: "c461be4b-e409-48fe-bcaf-6ba8e18ab312"). InnerVolumeSpecName "kube-api-access-2r2l6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:19.337010 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.336972 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8"] Oct 30 21:30:19.360641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.360609 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r2l6\" (UniqueName: \"kubernetes.io/projected/c461be4b-e409-48fe-bcaf-6ba8e18ab312-kube-api-access-2r2l6\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:19.390111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.388297 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9" event={"ID":"bde714d2-046b-443a-a5e6-f87c729e5ab0","Type":"ContainerStarted","Data":"79612912ae5b95fe7986631aeebf0e54d17d6626dc9e55385d92b55734e4a104"} Oct 30 21:30:19.393931 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.393163 4926 generic.go:356] "Generic (PLEG): container finished" podID="731eb7a0-c9e9-41f9-b245-0fe7a00be673" containerID="4c501f3819b8567eae77a41737f0eae4c992296f651bbc1a0b813299bf1f4cf6" exitCode=60 Oct 30 21:30:19.394752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.394369 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8" event={"ID":"731eb7a0-c9e9-41f9-b245-0fe7a00be673","Type":"ContainerDied","Data":"4c501f3819b8567eae77a41737f0eae4c992296f651bbc1a0b813299bf1f4cf6"} Oct 30 21:30:19.394752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.394460 4926 scope.go:122] "RemoveContainer" containerID="16529e09a3270761076d03523322758f43f467eeee987453ae987fca8d3657d9" Oct 30 21:30:19.403474 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.403443 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq" event={"ID":"fce0f54b-5f27-4637-8b5c-d6f2298873bb","Type":"ContainerStarted","Data":"74657909bbe9b980a307eb4d70923459467a21214deb5cd15c7bb47be695acd3"} Oct 30 21:30:19.409686 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.408341 4926 generic.go:356] "Generic (PLEG): container finished" podID="f17fd537-658d-4c4f-9aeb-a763d184c364" containerID="af90c2709ac2900d32a7435147c99776276fb4b4ae4d581be32fdad915b1a3d5" exitCode=60 Oct 30 21:30:19.409686 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.408464 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4" event={"ID":"f17fd537-658d-4c4f-9aeb-a763d184c364","Type":"ContainerDied","Data":"af90c2709ac2900d32a7435147c99776276fb4b4ae4d581be32fdad915b1a3d5"} Oct 30 21:30:19.409686 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.408883 4926 scope.go:122] "RemoveContainer" containerID="af90c2709ac2900d32a7435147c99776276fb4b4ae4d581be32fdad915b1a3d5" Oct 30 21:30:19.409686 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:19.409313 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4_odh-ai-gateway-infra(f17fd537-658d-4c4f-9aeb-a763d184c364)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4" podUID="f17fd537-658d-4c4f-9aeb-a763d184c364" Oct 30 21:30:19.423085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.422936 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj" event={"ID":"c461be4b-e409-48fe-bcaf-6ba8e18ab312","Type":"ContainerDied","Data":"a1c73588199c8d9a91a620bd9f93dcbdefea0191cb2400349d66b2088f842059"} Oct 30 21:30:19.425366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.423279 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj" Oct 30 21:30:19.428305 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.428233 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9" Oct 30 21:30:19.428473 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.428238 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9" event={"ID":"fd7f7c91-0516-476c-afc8-94130b2e3439","Type":"ContainerDied","Data":"3aa135ef51b2c3b573a3aea991aa1520e02af382f859f08a237b6bbdb2c42066"} Oct 30 21:30:19.429668 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.429649 4926 scope.go:122] "RemoveContainer" containerID="7ae5b8b41f62bd2dc3fc4b90234d7e820a3c4b140230b012ae4b71f583e06150" Oct 30 21:30:19.434174 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.434152 4926 generic.go:356] "Generic (PLEG): container finished" podID="6d846cc6-bbdd-456a-a049-e9d99773172d" containerID="b49204426ca44e188614c4889ad6356be236fdafb3b71dcfbb7fd6027e36c404" exitCode=60 Oct 30 21:30:19.434318 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.434290 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4" event={"ID":"6d846cc6-bbdd-456a-a049-e9d99773172d","Type":"ContainerDied","Data":"b49204426ca44e188614c4889ad6356be236fdafb3b71dcfbb7fd6027e36c404"} Oct 30 21:30:19.434803 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.434786 4926 scope.go:122] "RemoveContainer" containerID="b49204426ca44e188614c4889ad6356be236fdafb3b71dcfbb7fd6027e36c404" Oct 30 21:30:19.435207 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:19.435185 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4_odh-ai-gateway-infra(6d846cc6-bbdd-456a-a049-e9d99773172d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4" podUID="6d846cc6-bbdd-456a-a049-e9d99773172d" Oct 30 21:30:19.436861 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.436636 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6" event={"ID":"810d266f-62a8-4b0f-a146-89ed2e41df99","Type":"ContainerStarted","Data":"5ea93ceefb6951e82248495fd395442bde541ad176b410e9edec9d0fa5092484"} Oct 30 21:30:19.452620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.452599 4926 scope.go:122] "RemoveContainer" containerID="9e82baae24716dafa3d2eb828c55b7c0e5ab8438a2344e505b5fc451d6700828" Oct 30 21:30:19.477251 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.477220 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4"] Oct 30 21:30:19.487307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.487174 4926 scope.go:122] "RemoveContainer" containerID="89b3873d7684979ca950b9a68ece1041fbb881ed25448baa02e3fa471ec32956" Oct 30 21:30:19.524242 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.523600 4926 scope.go:122] "RemoveContainer" containerID="5151329cc5da7e64d088252663ae0bf1a64daef1a827bf605b2531c0cc504630" Oct 30 21:30:19.849733 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.849656 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4"] Oct 30 21:30:19.884603 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:19.881714 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9"] Oct 30 21:30:20.017066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:20.014881 4926 scope.go:122] "RemoveContainer" containerID="d9bb99512c4d52c3ac2c4895353bda862dafa2d376f89298f0b5f0171b79ada4" Oct 30 21:30:20.694191 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:20.673622 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:30:20.916618 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:20.916576 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Oct 30 21:30:20.956777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:20.956653 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Oct 30 21:30:21.064054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.042644 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" podUID="50ced360-7983-4b5a-8b03-35714cd87a9c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.159:8000/ready\": context deadline exceeded" Oct 30 21:30:21.162061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.064744 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" podUID="9c2bb6ce-be54-4671-a234-759effcd695c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.161:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:30:21.172751 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.117393 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:30:21.172959 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.137856 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" probeResult="failure" output="Get \"https://10.132.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:30:21.173511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.152176 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" podUID="f05d1479-4fa1-4640-a6f5-59f43084b401" containerName="main" probeResult="failure" output="Get \"https://10.132.0.158:8000/ready\": context deadline exceeded" Oct 30 21:30:21.260593 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.260493 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Oct 30 21:30:21.404148 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.403475 4926 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" podUID="9c2bb6ce-be54-4671-a234-759effcd695c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.161:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:30:21.412140 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.410533 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Oct 30 21:30:21.421064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.415138 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" podUID="6df032f2-153b-48f9-9e31-3ad79d10bcfc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.155:8000/ready\": context deadline exceeded" Oct 30 21:30:21.430634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.430085 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:30:21.434491 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.434465 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8" Oct 30 21:30:21.441715 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.441693 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889930-b6tr9"] Oct 30 21:30:21.456329 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.456201 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg" Oct 30 21:30:21.456968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.456948 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc" event={"ID":"d55b8ddb-6b3e-4f51-8281-f3209c63d949","Type":"ContainerStarted","Data":"4076b78c0a330c7a76b31df48a0125e76461165a8926a785426fc1c4e01f44d8"} Oct 30 21:30:21.457695 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.457567 4926 scope.go:122] "RemoveContainer" containerID="4076b78c0a330c7a76b31df48a0125e76461165a8926a785426fc1c4e01f44d8" Oct 30 21:30:21.457973 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:21.457951 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-lmlbc_odh-ai-gateway-infra(d55b8ddb-6b3e-4f51-8281-f3209c63d949)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc" podUID="d55b8ddb-6b3e-4f51-8281-f3209c63d949" Oct 30 21:30:21.483349 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.463918 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj"] Oct 30 21:30:21.483349 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.465191 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" event={"ID":"63543c7d-5e17-4248-82f2-b5b947e7aa4c","Type":"ContainerStarted","Data":"342dfef77f5547930a85b8d21d9fa0364d03e791d0bb44b8e3333e0a48d9e66f"} Oct 30 21:30:21.483349 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.465509 4926 scope.go:122] "RemoveContainer" containerID="342dfef77f5547930a85b8d21d9fa0364d03e791d0bb44b8e3333e0a48d9e66f" Oct 30 21:30:21.483349 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:21.465766 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6_odh-ai-gateway-infra(63543c7d-5e17-4248-82f2-b5b947e7aa4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" podUID="63543c7d-5e17-4248-82f2-b5b947e7aa4c" Oct 30 21:30:21.508049 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.505500 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889930-4b9nj"] Oct 30 21:30:21.508049 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.505928 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6" event={"ID":"810d266f-62a8-4b0f-a146-89ed2e41df99","Type":"ContainerDied","Data":"5ea93ceefb6951e82248495fd395442bde541ad176b410e9edec9d0fa5092484"} Oct 30 21:30:21.508049 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.505994 4926 scope.go:122] "RemoveContainer" containerID="f6019dd0da685ea4b454854fc35d7995182043f1a0afabcdc9d125bb28330750" Oct 30 21:30:21.508359 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.505760 4926 generic.go:356] "Generic (PLEG): container finished" podID="810d266f-62a8-4b0f-a146-89ed2e41df99" containerID="5ea93ceefb6951e82248495fd395442bde541ad176b410e9edec9d0fa5092484" exitCode=60 Oct 30 21:30:21.521298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.511387 4926 scope.go:122] "RemoveContainer" containerID="5ea93ceefb6951e82248495fd395442bde541ad176b410e9edec9d0fa5092484" Oct 30 21:30:21.521298 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:21.511926 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6_odh-ai-gateway-infra(810d266f-62a8-4b0f-a146-89ed2e41df99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6" podUID="810d266f-62a8-4b0f-a146-89ed2e41df99" Oct 30 21:30:21.521298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.512056 4926 generic.go:356] "Generic (PLEG): container finished" podID="bde714d2-046b-443a-a5e6-f87c729e5ab0" containerID="79612912ae5b95fe7986631aeebf0e54d17d6626dc9e55385d92b55734e4a104" exitCode=60 Oct 30 21:30:21.521298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.512137 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9" event={"ID":"bde714d2-046b-443a-a5e6-f87c729e5ab0","Type":"ContainerDied","Data":"79612912ae5b95fe7986631aeebf0e54d17d6626dc9e55385d92b55734e4a104"} Oct 30 21:30:21.521298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.512632 4926 scope.go:122] "RemoveContainer" containerID="79612912ae5b95fe7986631aeebf0e54d17d6626dc9e55385d92b55734e4a104" Oct 30 21:30:21.521298 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:21.513080 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9_odh-ai-gateway-infra(bde714d2-046b-443a-a5e6-f87c729e5ab0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9" podUID="bde714d2-046b-443a-a5e6-f87c729e5ab0" Oct 30 21:30:21.521298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.517075 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8" event={"ID":"731eb7a0-c9e9-41f9-b245-0fe7a00be673","Type":"ContainerDied","Data":"0cbe6fb2c2efe739f1fcab313a13f23810be8d057c51317c56010ea12a0b39de"} Oct 30 21:30:21.521298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.517154 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8" Oct 30 21:30:21.521298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.520520 4926 generic.go:356] "Generic (PLEG): container finished" podID="fce0f54b-5f27-4637-8b5c-d6f2298873bb" containerID="74657909bbe9b980a307eb4d70923459467a21214deb5cd15c7bb47be695acd3" exitCode=60 Oct 30 21:30:21.521298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.520630 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq" event={"ID":"fce0f54b-5f27-4637-8b5c-d6f2298873bb","Type":"ContainerDied","Data":"74657909bbe9b980a307eb4d70923459467a21214deb5cd15c7bb47be695acd3"} Oct 30 21:30:21.521298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.521086 4926 scope.go:122] "RemoveContainer" containerID="74657909bbe9b980a307eb4d70923459467a21214deb5cd15c7bb47be695acd3" Oct 30 21:30:21.521942 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:21.521423 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq_odh-ai-gateway-infra(fce0f54b-5f27-4637-8b5c-d6f2298873bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq" podUID="fce0f54b-5f27-4637-8b5c-d6f2298873bb" Oct 30 21:30:21.523834 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.523799 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" event={"ID":"495acf58-df75-4721-aaeb-fa3dcb9a3cd9","Type":"ContainerStarted","Data":"5e27e222dbdace580151924779bbe0c5c09c08646e1e05c17364caa34a35a774"} Oct 30 21:30:21.524464 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.524449 4926 scope.go:122] "RemoveContainer" containerID="5e27e222dbdace580151924779bbe0c5c09c08646e1e05c17364caa34a35a774" Oct 30 21:30:21.527895 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.527069 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9"] Oct 30 21:30:21.527895 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:21.527821 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv_odh-ai-gateway-infra(495acf58-df75-4721-aaeb-fa3dcb9a3cd9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" podUID="495acf58-df75-4721-aaeb-fa3dcb9a3cd9" Oct 30 21:30:21.546000 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.545714 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" event={"ID":"f5d36899-edf3-4bb0-a13b-39c36bf0ed68","Type":"ContainerStarted","Data":"33fae32726715c6221fb430bf416181bd5a9a6a8951d359c5c21974f67a01bd4"} Oct 30 21:30:21.546180 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.546157 4926 scope.go:122] "RemoveContainer" containerID="33fae32726715c6221fb430bf416181bd5a9a6a8951d359c5c21974f67a01bd4" Oct 30 21:30:21.547424 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.546453 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731eb7a0-c9e9-41f9-b245-0fe7a00be673-kube-api-access-shchh\" (UniqueName: \"kubernetes.io/projected/731eb7a0-c9e9-41f9-b245-0fe7a00be673-kube-api-access-shchh\") pod \"731eb7a0-c9e9-41f9-b245-0fe7a00be673\" (UID: \"731eb7a0-c9e9-41f9-b245-0fe7a00be673\") " Oct 30 21:30:21.547424 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.546538 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d922621f-a5d6-4854-8f0e-ba0ee21ade8e-kube-api-access-x5kkp\" (UniqueName: \"kubernetes.io/projected/d922621f-a5d6-4854-8f0e-ba0ee21ade8e-kube-api-access-x5kkp\") pod \"d922621f-a5d6-4854-8f0e-ba0ee21ade8e\" (UID: \"d922621f-a5d6-4854-8f0e-ba0ee21ade8e\") " Oct 30 21:30:21.547424 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:21.546632 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r_odh-ai-gateway-infra(f5d36899-edf3-4bb0-a13b-39c36bf0ed68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" podUID="f5d36899-edf3-4bb0-a13b-39c36bf0ed68" Oct 30 21:30:21.547424 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.546673 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d922621f-a5d6-4854-8f0e-ba0ee21ade8e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d922621f-a5d6-4854-8f0e-ba0ee21ade8e-maas-api-service-ca\") pod \"d922621f-a5d6-4854-8f0e-ba0ee21ade8e\" (UID: \"d922621f-a5d6-4854-8f0e-ba0ee21ade8e\") " Oct 30 21:30:21.547424 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.546764 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/731eb7a0-c9e9-41f9-b245-0fe7a00be673-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/731eb7a0-c9e9-41f9-b245-0fe7a00be673-maas-api-service-ca\") pod \"731eb7a0-c9e9-41f9-b245-0fe7a00be673\" (UID: \"731eb7a0-c9e9-41f9-b245-0fe7a00be673\") " Oct 30 21:30:21.549245 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.549201 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731eb7a0-c9e9-41f9-b245-0fe7a00be673-kube-api-access-shchh" pod "731eb7a0-c9e9-41f9-b245-0fe7a00be673" (UID: "731eb7a0-c9e9-41f9-b245-0fe7a00be673"). InnerVolumeSpecName "kube-api-access-shchh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:21.551562 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.549723 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/731eb7a0-c9e9-41f9-b245-0fe7a00be673-maas-api-service-ca" pod "731eb7a0-c9e9-41f9-b245-0fe7a00be673" (UID: "731eb7a0-c9e9-41f9-b245-0fe7a00be673"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:21.551562 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.549783 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d922621f-a5d6-4854-8f0e-ba0ee21ade8e-maas-api-service-ca" pod "d922621f-a5d6-4854-8f0e-ba0ee21ade8e" (UID: "d922621f-a5d6-4854-8f0e-ba0ee21ade8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:21.551562 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.550632 4926 scope.go:122] "RemoveContainer" containerID="3c091fd1ecd2c8ebba43baf1f7776abd134bad43211b1ead4d6b787b329e52ba" Oct 30 21:30:21.563684 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.563653 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq"] Oct 30 21:30:21.605507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.605467 4926 scope.go:122] "RemoveContainer" containerID="4c501f3819b8567eae77a41737f0eae4c992296f651bbc1a0b813299bf1f4cf6" Oct 30 21:30:21.609235 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.609109 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d922621f-a5d6-4854-8f0e-ba0ee21ade8e-kube-api-access-x5kkp" pod "d922621f-a5d6-4854-8f0e-ba0ee21ade8e" (UID: "d922621f-a5d6-4854-8f0e-ba0ee21ade8e"). InnerVolumeSpecName "kube-api-access-x5kkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:21.614688 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.614667 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6"] Oct 30 21:30:21.625350 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.624756 4926 scope.go:122] "RemoveContainer" containerID="22bc22dac5ddf8a3084da6a7755feaeb397fd1cb4b8080eb8aa40e1753a94c3d" Oct 30 21:30:21.653389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.650870 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/731eb7a0-c9e9-41f9-b245-0fe7a00be673-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:21.653389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.650904 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shchh\" (UniqueName: \"kubernetes.io/projected/731eb7a0-c9e9-41f9-b245-0fe7a00be673-kube-api-access-shchh\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:21.653389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.650921 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5kkp\" (UniqueName: \"kubernetes.io/projected/d922621f-a5d6-4854-8f0e-ba0ee21ade8e-kube-api-access-x5kkp\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:21.653389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.650936 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d922621f-a5d6-4854-8f0e-ba0ee21ade8e-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:21.753579 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.753495 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw"] Oct 30 21:30:21.754461 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.754412 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" podUID="217454ad-7acd-4b25-b91f-885bc464b6c5" containerName="istio-proxy" containerID="cri-o://301f5a8ab1e97447872c54d0c6049409b4330f445905eb6671ca868a5fec20ad" gracePeriod=30 Oct 30 21:30:21.854332 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.854305 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4" Oct 30 21:30:21.947996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.947971 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4" Oct 30 21:30:21.957432 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.957403 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f17fd537-658d-4c4f-9aeb-a763d184c364-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17fd537-658d-4c4f-9aeb-a763d184c364-maas-api-service-ca\") pod \"f17fd537-658d-4c4f-9aeb-a763d184c364\" (UID: \"f17fd537-658d-4c4f-9aeb-a763d184c364\") " Oct 30 21:30:21.957656 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.957633 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f17fd537-658d-4c4f-9aeb-a763d184c364-kube-api-access-2b5f5\" (UniqueName: \"kubernetes.io/projected/f17fd537-658d-4c4f-9aeb-a763d184c364-kube-api-access-2b5f5\") pod \"f17fd537-658d-4c4f-9aeb-a763d184c364\" (UID: \"f17fd537-658d-4c4f-9aeb-a763d184c364\") " Oct 30 21:30:21.957778 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.957757 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f17fd537-658d-4c4f-9aeb-a763d184c364-maas-api-service-ca" pod "f17fd537-658d-4c4f-9aeb-a763d184c364" (UID: "f17fd537-658d-4c4f-9aeb-a763d184c364"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:21.958462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.958442 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f17fd537-658d-4c4f-9aeb-a763d184c364-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:21.960314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:21.960289 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f17fd537-658d-4c4f-9aeb-a763d184c364-kube-api-access-2b5f5" pod "f17fd537-658d-4c4f-9aeb-a763d184c364" (UID: "f17fd537-658d-4c4f-9aeb-a763d184c364"). InnerVolumeSpecName "kube-api-access-2b5f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:22.060009 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.059971 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d846cc6-bbdd-456a-a049-e9d99773172d-kube-api-access-c2phj\" (UniqueName: \"kubernetes.io/projected/6d846cc6-bbdd-456a-a049-e9d99773172d-kube-api-access-c2phj\") pod \"6d846cc6-bbdd-456a-a049-e9d99773172d\" (UID: \"6d846cc6-bbdd-456a-a049-e9d99773172d\") " Oct 30 21:30:22.060203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.060163 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d846cc6-bbdd-456a-a049-e9d99773172d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d846cc6-bbdd-456a-a049-e9d99773172d-maas-api-service-ca\") pod \"6d846cc6-bbdd-456a-a049-e9d99773172d\" (UID: \"6d846cc6-bbdd-456a-a049-e9d99773172d\") " Oct 30 21:30:22.060533 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.060508 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d846cc6-bbdd-456a-a049-e9d99773172d-maas-api-service-ca" pod "6d846cc6-bbdd-456a-a049-e9d99773172d" (UID: "6d846cc6-bbdd-456a-a049-e9d99773172d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:22.061397 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.061370 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b5f5\" (UniqueName: \"kubernetes.io/projected/f17fd537-658d-4c4f-9aeb-a763d184c364-kube-api-access-2b5f5\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:22.061501 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.061401 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d846cc6-bbdd-456a-a049-e9d99773172d-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:22.062377 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.062356 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d846cc6-bbdd-456a-a049-e9d99773172d-kube-api-access-c2phj" pod "6d846cc6-bbdd-456a-a049-e9d99773172d" (UID: "6d846cc6-bbdd-456a-a049-e9d99773172d"). InnerVolumeSpecName "kube-api-access-c2phj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:22.162448 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.162368 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8"] Oct 30 21:30:22.163088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.163064 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2phj\" (UniqueName: \"kubernetes.io/projected/6d846cc6-bbdd-456a-a049-e9d99773172d-kube-api-access-c2phj\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:22.200961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.200928 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-70d3a9-29889930-snsb8"] Oct 30 21:30:22.558732 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.558702 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4" Oct 30 21:30:22.558998 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.558693 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4" event={"ID":"f17fd537-658d-4c4f-9aeb-a763d184c364","Type":"ContainerDied","Data":"dce1b494793709bab0b66803cc37378cba9ff12bdcdeb098444a27e3fdf71a0d"} Oct 30 21:30:22.558998 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.558899 4926 scope.go:122] "RemoveContainer" containerID="af90c2709ac2900d32a7435147c99776276fb4b4ae4d581be32fdad915b1a3d5" Oct 30 21:30:22.561423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.561400 4926 generic.go:356] "Generic (PLEG): container finished" podID="d55b8ddb-6b3e-4f51-8281-f3209c63d949" containerID="4076b78c0a330c7a76b31df48a0125e76461165a8926a785426fc1c4e01f44d8" exitCode=60 Oct 30 21:30:22.561546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.561482 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc" event={"ID":"d55b8ddb-6b3e-4f51-8281-f3209c63d949","Type":"ContainerDied","Data":"4076b78c0a330c7a76b31df48a0125e76461165a8926a785426fc1c4e01f44d8"} Oct 30 21:30:22.562067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.562006 4926 scope.go:122] "RemoveContainer" containerID="4076b78c0a330c7a76b31df48a0125e76461165a8926a785426fc1c4e01f44d8" Oct 30 21:30:22.562443 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:22.562415 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889930-lmlbc_odh-ai-gateway-infra(d55b8ddb-6b3e-4f51-8281-f3209c63d949)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc" podUID="d55b8ddb-6b3e-4f51-8281-f3209c63d949" Oct 30 21:30:22.563458 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.563436 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4" event={"ID":"6d846cc6-bbdd-456a-a049-e9d99773172d","Type":"ContainerDied","Data":"beba957622d7e596338ddfe08ac2f7f44086ce7f8488f8f294f460905167c87e"} Oct 30 21:30:22.563536 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.563459 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4" Oct 30 21:30:22.605589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.605559 4926 scope.go:122] "RemoveContainer" containerID="e156e111879c18643c59b38af35a0dc21f5f958b7947d0534a78c93014a5531c" Oct 30 21:30:22.605795 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.605765 4926 generic.go:356] "Generic (PLEG): container finished" podID="63543c7d-5e17-4248-82f2-b5b947e7aa4c" containerID="342dfef77f5547930a85b8d21d9fa0364d03e791d0bb44b8e3333e0a48d9e66f" exitCode=60 Oct 30 21:30:22.605889 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.605798 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" event={"ID":"63543c7d-5e17-4248-82f2-b5b947e7aa4c","Type":"ContainerDied","Data":"342dfef77f5547930a85b8d21d9fa0364d03e791d0bb44b8e3333e0a48d9e66f"} Oct 30 21:30:22.606301 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.606277 4926 scope.go:122] "RemoveContainer" containerID="342dfef77f5547930a85b8d21d9fa0364d03e791d0bb44b8e3333e0a48d9e66f" Oct 30 21:30:22.606682 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:22.606591 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6_odh-ai-gateway-infra(63543c7d-5e17-4248-82f2-b5b947e7aa4c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" podUID="63543c7d-5e17-4248-82f2-b5b947e7aa4c" Oct 30 21:30:22.613419 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.613397 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg" event={"ID":"d922621f-a5d6-4854-8f0e-ba0ee21ade8e","Type":"ContainerDied","Data":"f4383edc38fc9eb9e989e08c4cad314e4d9b31c9e7a0785bc92782eae89d4847"} Oct 30 21:30:22.613907 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.613889 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg" Oct 30 21:30:22.622585 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.621708 4926 generic.go:356] "Generic (PLEG): container finished" podID="495acf58-df75-4721-aaeb-fa3dcb9a3cd9" containerID="5e27e222dbdace580151924779bbe0c5c09c08646e1e05c17364caa34a35a774" exitCode=60 Oct 30 21:30:22.622585 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.621780 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" event={"ID":"495acf58-df75-4721-aaeb-fa3dcb9a3cd9","Type":"ContainerDied","Data":"5e27e222dbdace580151924779bbe0c5c09c08646e1e05c17364caa34a35a774"} Oct 30 21:30:22.622585 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.622214 4926 scope.go:122] "RemoveContainer" containerID="5e27e222dbdace580151924779bbe0c5c09c08646e1e05c17364caa34a35a774" Oct 30 21:30:22.622585 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:22.622520 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv_odh-ai-gateway-infra(495acf58-df75-4721-aaeb-fa3dcb9a3cd9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" podUID="495acf58-df75-4721-aaeb-fa3dcb9a3cd9" Oct 30 21:30:22.626082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.625307 4926 generic.go:356] "Generic (PLEG): container finished" podID="f5d36899-edf3-4bb0-a13b-39c36bf0ed68" containerID="33fae32726715c6221fb430bf416181bd5a9a6a8951d359c5c21974f67a01bd4" exitCode=60 Oct 30 21:30:22.626082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.625420 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" event={"ID":"f5d36899-edf3-4bb0-a13b-39c36bf0ed68","Type":"ContainerDied","Data":"33fae32726715c6221fb430bf416181bd5a9a6a8951d359c5c21974f67a01bd4"} Oct 30 21:30:22.626082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.625781 4926 scope.go:122] "RemoveContainer" containerID="33fae32726715c6221fb430bf416181bd5a9a6a8951d359c5c21974f67a01bd4" Oct 30 21:30:22.626082 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:22.626023 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r_odh-ai-gateway-infra(f5d36899-edf3-4bb0-a13b-39c36bf0ed68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" podUID="f5d36899-edf3-4bb0-a13b-39c36bf0ed68" Oct 30 21:30:22.628338 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.628313 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" event={"ID":"0fe4049a-063d-4990-8524-88c95c2d8862","Type":"ContainerStarted","Data":"5902918f2f7ea25d67ebfb66d3745e5e9da4e0b4ee0c22db680abc704ada2f81"} Oct 30 21:30:22.628668 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.628650 4926 scope.go:122] "RemoveContainer" containerID="5902918f2f7ea25d67ebfb66d3745e5e9da4e0b4ee0c22db680abc704ada2f81" Oct 30 21:30:22.628910 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:22.628889 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28_odh-ai-gateway-infra(0fe4049a-063d-4990-8524-88c95c2d8862)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" podUID="0fe4049a-063d-4990-8524-88c95c2d8862" Oct 30 21:30:22.706990 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.706894 4926 scope.go:122] "RemoveContainer" containerID="b49204426ca44e188614c4889ad6356be236fdafb3b71dcfbb7fd6027e36c404" Oct 30 21:30:22.721865 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.721841 4926 scope.go:122] "RemoveContainer" containerID="9275fab04d8ed86827ce84b220682141d3bbe0315748b9c8d8318d2dbd4e2b0c" Oct 30 21:30:22.752739 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.752698 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6"] Oct 30 21:30:22.794176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.773463 4926 scope.go:122] "RemoveContainer" containerID="7423efedcd8a3b136fef7f6c4ca8304045012844ffe518be73bd28f98f7dccbb" Oct 30 21:30:22.829260 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.826124 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc"] Oct 30 21:30:22.830146 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.830117 4926 scope.go:122] "RemoveContainer" containerID="c16157b7b56e36ac6c186eb4404beb0736686408a3c87fba717fc2c65e6bf0ae" Oct 30 21:30:22.859442 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.859413 4926 scope.go:122] "RemoveContainer" containerID="6147dde711b6adf97bc900cb4178581e1f20ee2e5be7f8ca8a2375d1d923d68c" Oct 30 21:30:22.933411 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.933340 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r"] Oct 30 21:30:22.946066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:22.945920 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6" Oct 30 21:30:23.032924 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.032893 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9" Oct 30 21:30:23.033122 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.033103 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="731eb7a0-c9e9-41f9-b245-0fe7a00be673" path="/var/lib/kubelet/pods/731eb7a0-c9e9-41f9-b245-0fe7a00be673/volumes" Oct 30 21:30:23.033640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.033619 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c461be4b-e409-48fe-bcaf-6ba8e18ab312" path="/var/lib/kubelet/pods/c461be4b-e409-48fe-bcaf-6ba8e18ab312/volumes" Oct 30 21:30:23.035275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.034070 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd7f7c91-0516-476c-afc8-94130b2e3439" path="/var/lib/kubelet/pods/fd7f7c91-0516-476c-afc8-94130b2e3439/volumes" Oct 30 21:30:23.103895 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.103866 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv"] Oct 30 21:30:23.116605 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.116578 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq" Oct 30 21:30:23.117366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.117340 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/810d266f-62a8-4b0f-a146-89ed2e41df99-kube-api-access-wmhvz\" (UniqueName: \"kubernetes.io/projected/810d266f-62a8-4b0f-a146-89ed2e41df99-kube-api-access-wmhvz\") pod \"810d266f-62a8-4b0f-a146-89ed2e41df99\" (UID: \"810d266f-62a8-4b0f-a146-89ed2e41df99\") " Oct 30 21:30:23.117530 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.117510 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/810d266f-62a8-4b0f-a146-89ed2e41df99-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/810d266f-62a8-4b0f-a146-89ed2e41df99-maas-api-service-ca\") pod \"810d266f-62a8-4b0f-a146-89ed2e41df99\" (UID: \"810d266f-62a8-4b0f-a146-89ed2e41df99\") " Oct 30 21:30:23.118483 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.118459 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/810d266f-62a8-4b0f-a146-89ed2e41df99-maas-api-service-ca" pod "810d266f-62a8-4b0f-a146-89ed2e41df99" (UID: "810d266f-62a8-4b0f-a146-89ed2e41df99"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:23.118934 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.118919 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/810d266f-62a8-4b0f-a146-89ed2e41df99-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:23.119815 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.119797 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/810d266f-62a8-4b0f-a146-89ed2e41df99-kube-api-access-wmhvz" pod "810d266f-62a8-4b0f-a146-89ed2e41df99" (UID: "810d266f-62a8-4b0f-a146-89ed2e41df99"). InnerVolumeSpecName "kube-api-access-wmhvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:23.211164 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.211123 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4"] Oct 30 21:30:23.217119 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.217090 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889930-8t9p4"] Oct 30 21:30:23.219552 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.219531 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bde714d2-046b-443a-a5e6-f87c729e5ab0-kube-api-access-qqmlh\" (UniqueName: \"kubernetes.io/projected/bde714d2-046b-443a-a5e6-f87c729e5ab0-kube-api-access-qqmlh\") pod \"bde714d2-046b-443a-a5e6-f87c729e5ab0\" (UID: \"bde714d2-046b-443a-a5e6-f87c729e5ab0\") " Oct 30 21:30:23.219762 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.219660 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bde714d2-046b-443a-a5e6-f87c729e5ab0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde714d2-046b-443a-a5e6-f87c729e5ab0-maas-api-service-ca\") pod \"bde714d2-046b-443a-a5e6-f87c729e5ab0\" (UID: \"bde714d2-046b-443a-a5e6-f87c729e5ab0\") " Oct 30 21:30:23.219998 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.219858 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fce0f54b-5f27-4637-8b5c-d6f2298873bb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce0f54b-5f27-4637-8b5c-d6f2298873bb-maas-api-service-ca\") pod \"fce0f54b-5f27-4637-8b5c-d6f2298873bb\" (UID: \"fce0f54b-5f27-4637-8b5c-d6f2298873bb\") " Oct 30 21:30:23.219998 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.219973 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fce0f54b-5f27-4637-8b5c-d6f2298873bb-kube-api-access-ns466\" (UniqueName: \"kubernetes.io/projected/fce0f54b-5f27-4637-8b5c-d6f2298873bb-kube-api-access-ns466\") pod \"fce0f54b-5f27-4637-8b5c-d6f2298873bb\" (UID: \"fce0f54b-5f27-4637-8b5c-d6f2298873bb\") " Oct 30 21:30:23.221201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.221177 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmhvz\" (UniqueName: \"kubernetes.io/projected/810d266f-62a8-4b0f-a146-89ed2e41df99-kube-api-access-wmhvz\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:23.221509 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.221479 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fce0f54b-5f27-4637-8b5c-d6f2298873bb-maas-api-service-ca" pod "fce0f54b-5f27-4637-8b5c-d6f2298873bb" (UID: "fce0f54b-5f27-4637-8b5c-d6f2298873bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:23.221693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.221674 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bde714d2-046b-443a-a5e6-f87c729e5ab0-maas-api-service-ca" pod "bde714d2-046b-443a-a5e6-f87c729e5ab0" (UID: "bde714d2-046b-443a-a5e6-f87c729e5ab0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:23.222447 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.222424 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce0f54b-5f27-4637-8b5c-d6f2298873bb-kube-api-access-ns466" pod "fce0f54b-5f27-4637-8b5c-d6f2298873bb" (UID: "fce0f54b-5f27-4637-8b5c-d6f2298873bb"). InnerVolumeSpecName "kube-api-access-ns466". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:23.224252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.224230 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bde714d2-046b-443a-a5e6-f87c729e5ab0-kube-api-access-qqmlh" pod "bde714d2-046b-443a-a5e6-f87c729e5ab0" (UID: "bde714d2-046b-443a-a5e6-f87c729e5ab0"). InnerVolumeSpecName "kube-api-access-qqmlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:23.252262 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.252232 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4"] Oct 30 21:30:23.256798 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.256772 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-093daa-29889930-xxfq4"] Oct 30 21:30:23.323642 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.323606 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns466\" (UniqueName: \"kubernetes.io/projected/fce0f54b-5f27-4637-8b5c-d6f2298873bb-kube-api-access-ns466\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:23.323642 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.323641 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqmlh\" (UniqueName: \"kubernetes.io/projected/bde714d2-046b-443a-a5e6-f87c729e5ab0-kube-api-access-qqmlh\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:23.323907 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.323656 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bde714d2-046b-443a-a5e6-f87c729e5ab0-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:23.323907 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.323669 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fce0f54b-5f27-4637-8b5c-d6f2298873bb-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:23.341135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.341100 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg"] Oct 30 21:30:23.346074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.346010 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889930-zrnvg"] Oct 30 21:30:23.645723 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.645695 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6" Oct 30 21:30:23.645870 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.645694 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6" event={"ID":"810d266f-62a8-4b0f-a146-89ed2e41df99","Type":"ContainerDied","Data":"c9168aaa1d70354b1794e65bdf184fd3e419dbcc9488576317e91aeeae403d2f"} Oct 30 21:30:23.645942 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.645870 4926 scope.go:122] "RemoveContainer" containerID="5ea93ceefb6951e82248495fd395442bde541ad176b410e9edec9d0fa5092484" Oct 30 21:30:23.647224 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.647200 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9" Oct 30 21:30:23.647356 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.647284 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9" event={"ID":"bde714d2-046b-443a-a5e6-f87c729e5ab0","Type":"ContainerDied","Data":"a7a12e7ef51f35b8f3c49dec9f0a8b5bd656190280be3c92e913b61690837280"} Oct 30 21:30:23.650691 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.650673 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq" Oct 30 21:30:23.650828 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.650701 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq" event={"ID":"fce0f54b-5f27-4637-8b5c-d6f2298873bb","Type":"ContainerDied","Data":"cd761a62a9bc014e15361d5734b51baa474c8d30c2457b7d9e28f7442498ebe7"} Oct 30 21:30:23.688369 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.687539 4926 scope.go:122] "RemoveContainer" containerID="79612912ae5b95fe7986631aeebf0e54d17d6626dc9e55385d92b55734e4a104" Oct 30 21:30:23.707459 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.707428 4926 generic.go:356] "Generic (PLEG): container finished" podID="0fe4049a-063d-4990-8524-88c95c2d8862" containerID="5902918f2f7ea25d67ebfb66d3745e5e9da4e0b4ee0c22db680abc704ada2f81" exitCode=60 Oct 30 21:30:23.707662 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.707628 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" event={"ID":"0fe4049a-063d-4990-8524-88c95c2d8862","Type":"ContainerDied","Data":"5902918f2f7ea25d67ebfb66d3745e5e9da4e0b4ee0c22db680abc704ada2f81"} Oct 30 21:30:23.712561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.712540 4926 scope.go:122] "RemoveContainer" containerID="5902918f2f7ea25d67ebfb66d3745e5e9da4e0b4ee0c22db680abc704ada2f81" Oct 30 21:30:23.713532 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:23.713495 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28_odh-ai-gateway-infra(0fe4049a-063d-4990-8524-88c95c2d8862)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" podUID="0fe4049a-063d-4990-8524-88c95c2d8862" Oct 30 21:30:23.741390 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.741360 4926 scope.go:122] "RemoveContainer" containerID="74657909bbe9b980a307eb4d70923459467a21214deb5cd15c7bb47be695acd3" Oct 30 21:30:23.768481 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.768367 4926 scope.go:122] "RemoveContainer" containerID="d9bb99512c4d52c3ac2c4895353bda862dafa2d376f89298f0b5f0171b79ada4" Oct 30 21:30:23.823132 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.822540 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq"] Oct 30 21:30:23.858026 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.857321 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-290ede-29889930-2rhsq"] Oct 30 21:30:23.910924 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.910842 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28"] Oct 30 21:30:23.946509 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.946447 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6"] Oct 30 21:30:23.954301 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.954214 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-67f8e0-29889930-8xpm6"] Oct 30 21:30:23.962903 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:23.962883 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" Oct 30 21:30:24.109265 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.109214 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9"] Oct 30 21:30:24.132276 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.132231 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1eab65-29889930-7str9"] Oct 30 21:30:24.133916 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.133862 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63543c7d-5e17-4248-82f2-b5b947e7aa4c-kube-api-access-rfcmv\" (UniqueName: \"kubernetes.io/projected/63543c7d-5e17-4248-82f2-b5b947e7aa4c-kube-api-access-rfcmv\") pod \"63543c7d-5e17-4248-82f2-b5b947e7aa4c\" (UID: \"63543c7d-5e17-4248-82f2-b5b947e7aa4c\") " Oct 30 21:30:24.134078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.133934 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63543c7d-5e17-4248-82f2-b5b947e7aa4c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63543c7d-5e17-4248-82f2-b5b947e7aa4c-maas-api-service-ca\") pod \"63543c7d-5e17-4248-82f2-b5b947e7aa4c\" (UID: \"63543c7d-5e17-4248-82f2-b5b947e7aa4c\") " Oct 30 21:30:24.136472 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.136439 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63543c7d-5e17-4248-82f2-b5b947e7aa4c-maas-api-service-ca" pod "63543c7d-5e17-4248-82f2-b5b947e7aa4c" (UID: "63543c7d-5e17-4248-82f2-b5b947e7aa4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:24.185650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.185594 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63543c7d-5e17-4248-82f2-b5b947e7aa4c-kube-api-access-rfcmv" pod "63543c7d-5e17-4248-82f2-b5b947e7aa4c" (UID: "63543c7d-5e17-4248-82f2-b5b947e7aa4c"). InnerVolumeSpecName "kube-api-access-rfcmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:24.250174 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.249570 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfcmv\" (UniqueName: \"kubernetes.io/projected/63543c7d-5e17-4248-82f2-b5b947e7aa4c-kube-api-access-rfcmv\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:24.250174 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.249607 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63543c7d-5e17-4248-82f2-b5b947e7aa4c-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:24.448846 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.448822 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" Oct 30 21:30:24.452259 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.452239 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" Oct 30 21:30:24.454791 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.454768 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc" Oct 30 21:30:24.584341 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.584297 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5d36899-edf3-4bb0-a13b-39c36bf0ed68-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d36899-edf3-4bb0-a13b-39c36bf0ed68-maas-api-service-ca\") pod \"f5d36899-edf3-4bb0-a13b-39c36bf0ed68\" (UID: \"f5d36899-edf3-4bb0-a13b-39c36bf0ed68\") " Oct 30 21:30:24.584900 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.584856 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495acf58-df75-4721-aaeb-fa3dcb9a3cd9-kube-api-access-t8xhn\" (UniqueName: \"kubernetes.io/projected/495acf58-df75-4721-aaeb-fa3dcb9a3cd9-kube-api-access-t8xhn\") pod \"495acf58-df75-4721-aaeb-fa3dcb9a3cd9\" (UID: \"495acf58-df75-4721-aaeb-fa3dcb9a3cd9\") " Oct 30 21:30:24.585500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.585478 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/495acf58-df75-4721-aaeb-fa3dcb9a3cd9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/495acf58-df75-4721-aaeb-fa3dcb9a3cd9-maas-api-service-ca\") pod \"495acf58-df75-4721-aaeb-fa3dcb9a3cd9\" (UID: \"495acf58-df75-4721-aaeb-fa3dcb9a3cd9\") " Oct 30 21:30:24.585823 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.585763 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d55b8ddb-6b3e-4f51-8281-f3209c63d949-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55b8ddb-6b3e-4f51-8281-f3209c63d949-maas-api-service-ca\") pod \"d55b8ddb-6b3e-4f51-8281-f3209c63d949\" (UID: \"d55b8ddb-6b3e-4f51-8281-f3209c63d949\") " Oct 30 21:30:24.586123 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.586096 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d55b8ddb-6b3e-4f51-8281-f3209c63d949-kube-api-access-fcrkp\" (UniqueName: \"kubernetes.io/projected/d55b8ddb-6b3e-4f51-8281-f3209c63d949-kube-api-access-fcrkp\") pod \"d55b8ddb-6b3e-4f51-8281-f3209c63d949\" (UID: \"d55b8ddb-6b3e-4f51-8281-f3209c63d949\") " Oct 30 21:30:24.586464 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.586434 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5d36899-edf3-4bb0-a13b-39c36bf0ed68-kube-api-access-v5l69\" (UniqueName: \"kubernetes.io/projected/f5d36899-edf3-4bb0-a13b-39c36bf0ed68-kube-api-access-v5l69\") pod \"f5d36899-edf3-4bb0-a13b-39c36bf0ed68\" (UID: \"f5d36899-edf3-4bb0-a13b-39c36bf0ed68\") " Oct 30 21:30:24.590648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.589334 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d55b8ddb-6b3e-4f51-8281-f3209c63d949-maas-api-service-ca" pod "d55b8ddb-6b3e-4f51-8281-f3209c63d949" (UID: "d55b8ddb-6b3e-4f51-8281-f3209c63d949"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:24.590648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.589898 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/495acf58-df75-4721-aaeb-fa3dcb9a3cd9-maas-api-service-ca" pod "495acf58-df75-4721-aaeb-fa3dcb9a3cd9" (UID: "495acf58-df75-4721-aaeb-fa3dcb9a3cd9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:24.591145 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.590966 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5d36899-edf3-4bb0-a13b-39c36bf0ed68-maas-api-service-ca" pod "f5d36899-edf3-4bb0-a13b-39c36bf0ed68" (UID: "f5d36899-edf3-4bb0-a13b-39c36bf0ed68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:24.644252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.619206 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5d36899-edf3-4bb0-a13b-39c36bf0ed68-kube-api-access-v5l69" pod "f5d36899-edf3-4bb0-a13b-39c36bf0ed68" (UID: "f5d36899-edf3-4bb0-a13b-39c36bf0ed68"). InnerVolumeSpecName "kube-api-access-v5l69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:24.644252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.620843 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d55b8ddb-6b3e-4f51-8281-f3209c63d949-kube-api-access-fcrkp" pod "d55b8ddb-6b3e-4f51-8281-f3209c63d949" (UID: "d55b8ddb-6b3e-4f51-8281-f3209c63d949"). InnerVolumeSpecName "kube-api-access-fcrkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:24.647509 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.647442 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495acf58-df75-4721-aaeb-fa3dcb9a3cd9-kube-api-access-t8xhn" pod "495acf58-df75-4721-aaeb-fa3dcb9a3cd9" (UID: "495acf58-df75-4721-aaeb-fa3dcb9a3cd9"). InnerVolumeSpecName "kube-api-access-t8xhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:24.695939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.692507 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5l69\" (UniqueName: \"kubernetes.io/projected/f5d36899-edf3-4bb0-a13b-39c36bf0ed68-kube-api-access-v5l69\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:24.695939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.692552 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5d36899-edf3-4bb0-a13b-39c36bf0ed68-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:24.695939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.692568 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8xhn\" (UniqueName: \"kubernetes.io/projected/495acf58-df75-4721-aaeb-fa3dcb9a3cd9-kube-api-access-t8xhn\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:24.695939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.692582 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/495acf58-df75-4721-aaeb-fa3dcb9a3cd9-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:24.695939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.692598 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d55b8ddb-6b3e-4f51-8281-f3209c63d949-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:24.695939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.692612 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcrkp\" (UniqueName: \"kubernetes.io/projected/d55b8ddb-6b3e-4f51-8281-f3209c63d949-kube-api-access-fcrkp\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:24.741810 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.741775 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" event={"ID":"495acf58-df75-4721-aaeb-fa3dcb9a3cd9","Type":"ContainerDied","Data":"5bc106f185b03c4e6b7d2dfcbd8b0d75351ebe3e1d3e4aaeb6b64ed2fc9e6415"} Oct 30 21:30:24.741997 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.741832 4926 scope.go:122] "RemoveContainer" containerID="5e27e222dbdace580151924779bbe0c5c09c08646e1e05c17364caa34a35a774" Oct 30 21:30:24.741997 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.741904 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv" Oct 30 21:30:24.749251 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.749223 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" Oct 30 21:30:24.749436 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.749302 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r" event={"ID":"f5d36899-edf3-4bb0-a13b-39c36bf0ed68","Type":"ContainerDied","Data":"2c8e8f8c309c509bb754789757ea90cce7ac456ed4256bd06f83947c6597df2e"} Oct 30 21:30:24.811168 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.811142 4926 scope.go:122] "RemoveContainer" containerID="33fae32726715c6221fb430bf416181bd5a9a6a8951d359c5c21974f67a01bd4" Oct 30 21:30:24.811700 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.811673 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc" event={"ID":"d55b8ddb-6b3e-4f51-8281-f3209c63d949","Type":"ContainerDied","Data":"d4f5ff07ee094cbdbd85243b2983b8c22a1230629566fa37223174a7fd399be5"} Oct 30 21:30:24.811940 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.811922 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc" Oct 30 21:30:24.843724 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.842143 4926 generic.go:356] "Generic (PLEG): container finished" podID="87424b1f-d6bc-42f6-ad13-b85cc826043f" containerID="6fab60554de72477679321feab0725fd0923c88c86b71079abd8285f67784890" exitCode=0 Oct 30 21:30:24.843724 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.842298 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" event={"ID":"87424b1f-d6bc-42f6-ad13-b85cc826043f","Type":"ContainerDied","Data":"6fab60554de72477679321feab0725fd0923c88c86b71079abd8285f67784890"} Oct 30 21:30:24.846887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.844980 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" event={"ID":"63543c7d-5e17-4248-82f2-b5b947e7aa4c","Type":"ContainerDied","Data":"acef094367c547b9ac9b98afd23cd132ef0b48f0193bb5caadd3c30fa11817dc"} Oct 30 21:30:24.846887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.845098 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6" Oct 30 21:30:24.940094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.938754 4926 scope.go:122] "RemoveContainer" containerID="4076b78c0a330c7a76b31df48a0125e76461165a8926a785426fc1c4e01f44d8" Oct 30 21:30:24.965248 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:24.965110 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:30:25.029933 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.029903 4926 scope.go:122] "RemoveContainer" containerID="342dfef77f5547930a85b8d21d9fa0364d03e791d0bb44b8e3333e0a48d9e66f" Oct 30 21:30:25.039691 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.039657 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d846cc6-bbdd-456a-a049-e9d99773172d" path="/var/lib/kubelet/pods/6d846cc6-bbdd-456a-a049-e9d99773172d/volumes" Oct 30 21:30:25.040497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.040468 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="810d266f-62a8-4b0f-a146-89ed2e41df99" path="/var/lib/kubelet/pods/810d266f-62a8-4b0f-a146-89ed2e41df99/volumes" Oct 30 21:30:25.041137 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.041116 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bde714d2-046b-443a-a5e6-f87c729e5ab0" path="/var/lib/kubelet/pods/bde714d2-046b-443a-a5e6-f87c729e5ab0/volumes" Oct 30 21:30:25.042192 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.042170 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d922621f-a5d6-4854-8f0e-ba0ee21ade8e" path="/var/lib/kubelet/pods/d922621f-a5d6-4854-8f0e-ba0ee21ade8e/volumes" Oct 30 21:30:25.042808 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.042785 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f17fd537-658d-4c4f-9aeb-a763d184c364" path="/var/lib/kubelet/pods/f17fd537-658d-4c4f-9aeb-a763d184c364/volumes" Oct 30 21:30:25.043416 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.043395 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fce0f54b-5f27-4637-8b5c-d6f2298873bb" path="/var/lib/kubelet/pods/fce0f54b-5f27-4637-8b5c-d6f2298873bb/volumes" Oct 30 21:30:25.048057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.047851 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv"] Oct 30 21:30:25.048057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.047890 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889930-krrkv"] Oct 30 21:30:25.095803 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.095734 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc"] Oct 30 21:30:25.147314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.147283 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-credential-socket\") pod \"87424b1f-d6bc-42f6-ad13-b85cc826043f\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " Oct 30 21:30:25.190464 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.190431 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87424b1f-d6bc-42f6-ad13-b85cc826043f-kube-api-access-d5c2w\" (UniqueName: \"kubernetes.io/projected/87424b1f-d6bc-42f6-ad13-b85cc826043f-kube-api-access-d5c2w\") pod \"87424b1f-d6bc-42f6-ad13-b85cc826043f\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " Oct 30 21:30:25.191411 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.191384 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-token\" (UniqueName: \"kubernetes.io/projected/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-token\") pod \"87424b1f-d6bc-42f6-ad13-b85cc826043f\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " Oct 30 21:30:25.191687 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.191669 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87424b1f-d6bc-42f6-ad13-b85cc826043f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87424b1f-d6bc-42f6-ad13-b85cc826043f-istiod-ca-cert\") pod \"87424b1f-d6bc-42f6-ad13-b85cc826043f\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " Oct 30 21:30:25.191827 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.191812 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-workload-socket\") pod \"87424b1f-d6bc-42f6-ad13-b85cc826043f\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " Oct 30 21:30:25.192202 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.192178 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-podinfo\") pod \"87424b1f-d6bc-42f6-ad13-b85cc826043f\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " Oct 30 21:30:25.193474 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.193453 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-workload-certs\") pod \"87424b1f-d6bc-42f6-ad13-b85cc826043f\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " Oct 30 21:30:25.193739 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.193717 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-envoy\") pod \"87424b1f-d6bc-42f6-ad13-b85cc826043f\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " Oct 30 21:30:25.208299 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.208270 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-data\") pod \"87424b1f-d6bc-42f6-ad13-b85cc826043f\" (UID: \"87424b1f-d6bc-42f6-ad13-b85cc826043f\") " Oct 30 21:30:25.214837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.147939 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-credential-socket" pod "87424b1f-d6bc-42f6-ad13-b85cc826043f" (UID: "87424b1f-d6bc-42f6-ad13-b85cc826043f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:25.214837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.194317 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-workload-socket" pod "87424b1f-d6bc-42f6-ad13-b85cc826043f" (UID: "87424b1f-d6bc-42f6-ad13-b85cc826043f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:25.214837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.211270 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87424b1f-d6bc-42f6-ad13-b85cc826043f-istiod-ca-cert" pod "87424b1f-d6bc-42f6-ad13-b85cc826043f" (UID: "87424b1f-d6bc-42f6-ad13-b85cc826043f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:25.214837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.212184 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-workload-certs" pod "87424b1f-d6bc-42f6-ad13-b85cc826043f" (UID: "87424b1f-d6bc-42f6-ad13-b85cc826043f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:25.217909 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.217854 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-data" pod "87424b1f-d6bc-42f6-ad13-b85cc826043f" (UID: "87424b1f-d6bc-42f6-ad13-b85cc826043f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:25.227401 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.227383 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-podinfo" pod "87424b1f-d6bc-42f6-ad13-b85cc826043f" (UID: "87424b1f-d6bc-42f6-ad13-b85cc826043f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:30:25.228311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.228284 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-token" pod "87424b1f-d6bc-42f6-ad13-b85cc826043f" (UID: "87424b1f-d6bc-42f6-ad13-b85cc826043f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:25.228518 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.228505 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87424b1f-d6bc-42f6-ad13-b85cc826043f-kube-api-access-d5c2w" pod "87424b1f-d6bc-42f6-ad13-b85cc826043f" (UID: "87424b1f-d6bc-42f6-ad13-b85cc826043f"). InnerVolumeSpecName "kube-api-access-d5c2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:25.228651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.228632 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-envoy" pod "87424b1f-d6bc-42f6-ad13-b85cc826043f" (UID: "87424b1f-d6bc-42f6-ad13-b85cc826043f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:25.229170 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.229142 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889930-lmlbc"] Oct 30 21:30:25.237470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.237449 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r"] Oct 30 21:30:25.244736 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.244716 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889930-khg4r"] Oct 30 21:30:25.252341 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.252323 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6"] Oct 30 21:30:25.260252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.260227 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9477c-29889930-hqmf6"] Oct 30 21:30:25.321300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.321270 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:25.321498 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.321481 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5c2w\" (UniqueName: \"kubernetes.io/projected/87424b1f-d6bc-42f6-ad13-b85cc826043f-kube-api-access-d5c2w\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:25.321599 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.321586 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:25.321708 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.321697 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87424b1f-d6bc-42f6-ad13-b85cc826043f-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:25.321790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.321780 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:25.321871 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.321861 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:25.321948 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.321939 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:25.322026 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.322016 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:25.322125 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.322114 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87424b1f-d6bc-42f6-ad13-b85cc826043f-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:25.328490 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.328472 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" Oct 30 21:30:25.428307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.423753 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fe4049a-063d-4990-8524-88c95c2d8862-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fe4049a-063d-4990-8524-88c95c2d8862-maas-api-service-ca\") pod \"0fe4049a-063d-4990-8524-88c95c2d8862\" (UID: \"0fe4049a-063d-4990-8524-88c95c2d8862\") " Oct 30 21:30:25.507841 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.507754 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe4049a-063d-4990-8524-88c95c2d8862-kube-api-access-znkj9\" (UniqueName: \"kubernetes.io/projected/0fe4049a-063d-4990-8524-88c95c2d8862-kube-api-access-znkj9\") pod \"0fe4049a-063d-4990-8524-88c95c2d8862\" (UID: \"0fe4049a-063d-4990-8524-88c95c2d8862\") " Oct 30 21:30:25.512082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.506325 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe4049a-063d-4990-8524-88c95c2d8862-maas-api-service-ca" pod "0fe4049a-063d-4990-8524-88c95c2d8862" (UID: "0fe4049a-063d-4990-8524-88c95c2d8862"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:25.635404 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.561875 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe4049a-063d-4990-8524-88c95c2d8862-kube-api-access-znkj9" pod "0fe4049a-063d-4990-8524-88c95c2d8862" (UID: "0fe4049a-063d-4990-8524-88c95c2d8862"). InnerVolumeSpecName "kube-api-access-znkj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:25.736820 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.736754 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fe4049a-063d-4990-8524-88c95c2d8862-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:25.737271 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.737246 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znkj9\" (UniqueName: \"kubernetes.io/projected/0fe4049a-063d-4990-8524-88c95c2d8862-kube-api-access-znkj9\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:25.862548 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.862482 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" Oct 30 21:30:25.862548 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.862507 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28" event={"ID":"0fe4049a-063d-4990-8524-88c95c2d8862","Type":"ContainerDied","Data":"198932983fa93708461213ec05eeae28da3e2b063d973cd0f95bfda4fe093910"} Oct 30 21:30:25.863009 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.862609 4926 scope.go:122] "RemoveContainer" containerID="5902918f2f7ea25d67ebfb66d3745e5e9da4e0b4ee0c22db680abc704ada2f81" Oct 30 21:30:25.866879 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.866628 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" Oct 30 21:30:25.868274 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.868249 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78" event={"ID":"87424b1f-d6bc-42f6-ad13-b85cc826043f","Type":"ContainerDied","Data":"e075dea838309cf87a7e24fd8c55eb938043f7103ab444ad4594d21671e74e9c"} Oct 30 21:30:25.884217 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.884192 4926 scope.go:122] "RemoveContainer" containerID="6fab60554de72477679321feab0725fd0923c88c86b71079abd8285f67784890" Oct 30 21:30:25.956790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.956752 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28"] Oct 30 21:30:25.964482 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.964450 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-79e911-29889930-c2x28"] Oct 30 21:30:26.000568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:25.995202 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78"] Oct 30 21:30:26.006012 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:26.005982 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-xgv78"] Oct 30 21:30:26.902048 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:26.901461 4926 generic.go:356] "Generic (PLEG): container finished" podID="217454ad-7acd-4b25-b91f-885bc464b6c5" containerID="301f5a8ab1e97447872c54d0c6049409b4330f445905eb6671ca868a5fec20ad" exitCode=0 Oct 30 21:30:26.902048 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:26.901619 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" event={"ID":"217454ad-7acd-4b25-b91f-885bc464b6c5","Type":"ContainerDied","Data":"301f5a8ab1e97447872c54d0c6049409b4330f445905eb6671ca868a5fec20ad"} Oct 30 21:30:27.026384 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.026348 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fe4049a-063d-4990-8524-88c95c2d8862" path="/var/lib/kubelet/pods/0fe4049a-063d-4990-8524-88c95c2d8862/volumes" Oct 30 21:30:27.027124 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.027102 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="495acf58-df75-4721-aaeb-fa3dcb9a3cd9" path="/var/lib/kubelet/pods/495acf58-df75-4721-aaeb-fa3dcb9a3cd9/volumes" Oct 30 21:30:27.027858 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.027678 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63543c7d-5e17-4248-82f2-b5b947e7aa4c" path="/var/lib/kubelet/pods/63543c7d-5e17-4248-82f2-b5b947e7aa4c/volumes" Oct 30 21:30:27.028837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.028814 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87424b1f-d6bc-42f6-ad13-b85cc826043f" path="/var/lib/kubelet/pods/87424b1f-d6bc-42f6-ad13-b85cc826043f/volumes" Oct 30 21:30:27.029410 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.029391 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d55b8ddb-6b3e-4f51-8281-f3209c63d949" path="/var/lib/kubelet/pods/d55b8ddb-6b3e-4f51-8281-f3209c63d949/volumes" Oct 30 21:30:27.029946 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.029926 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5d36899-edf3-4bb0-a13b-39c36bf0ed68" path="/var/lib/kubelet/pods/f5d36899-edf3-4bb0-a13b-39c36bf0ed68/volumes" Oct 30 21:30:27.088845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.088816 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:30:27.169529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.167892 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/217454ad-7acd-4b25-b91f-885bc464b6c5-kube-api-access-t9njz\" (UniqueName: \"kubernetes.io/projected/217454ad-7acd-4b25-b91f-885bc464b6c5-kube-api-access-t9njz\") pod \"217454ad-7acd-4b25-b91f-885bc464b6c5\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " Oct 30 21:30:27.169529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.167956 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-credential-socket\") pod \"217454ad-7acd-4b25-b91f-885bc464b6c5\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " Oct 30 21:30:27.169529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.168066 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/217454ad-7acd-4b25-b91f-885bc464b6c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/217454ad-7acd-4b25-b91f-885bc464b6c5-istiod-ca-cert\") pod \"217454ad-7acd-4b25-b91f-885bc464b6c5\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " Oct 30 21:30:27.169529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.168130 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-workload-socket\") pod \"217454ad-7acd-4b25-b91f-885bc464b6c5\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " Oct 30 21:30:27.169529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.168254 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-token\" (UniqueName: \"kubernetes.io/projected/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-token\") pod \"217454ad-7acd-4b25-b91f-885bc464b6c5\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " Oct 30 21:30:27.169529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.168329 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-envoy\") pod \"217454ad-7acd-4b25-b91f-885bc464b6c5\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " Oct 30 21:30:27.169529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.168408 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-workload-certs\") pod \"217454ad-7acd-4b25-b91f-885bc464b6c5\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " Oct 30 21:30:27.169529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.168438 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-podinfo\") pod \"217454ad-7acd-4b25-b91f-885bc464b6c5\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " Oct 30 21:30:27.169529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.168503 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-data\") pod \"217454ad-7acd-4b25-b91f-885bc464b6c5\" (UID: \"217454ad-7acd-4b25-b91f-885bc464b6c5\") " Oct 30 21:30:27.170410 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.170373 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-workload-socket" pod "217454ad-7acd-4b25-b91f-885bc464b6c5" (UID: "217454ad-7acd-4b25-b91f-885bc464b6c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:27.171084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.171060 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-data" pod "217454ad-7acd-4b25-b91f-885bc464b6c5" (UID: "217454ad-7acd-4b25-b91f-885bc464b6c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:27.171602 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.171404 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-credential-socket" pod "217454ad-7acd-4b25-b91f-885bc464b6c5" (UID: "217454ad-7acd-4b25-b91f-885bc464b6c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:27.172078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.171846 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/217454ad-7acd-4b25-b91f-885bc464b6c5-istiod-ca-cert" pod "217454ad-7acd-4b25-b91f-885bc464b6c5" (UID: "217454ad-7acd-4b25-b91f-885bc464b6c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:30:27.172535 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.172218 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-workload-certs" pod "217454ad-7acd-4b25-b91f-885bc464b6c5" (UID: "217454ad-7acd-4b25-b91f-885bc464b6c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:27.174613 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.174579 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/217454ad-7acd-4b25-b91f-885bc464b6c5-kube-api-access-t9njz" pod "217454ad-7acd-4b25-b91f-885bc464b6c5" (UID: "217454ad-7acd-4b25-b91f-885bc464b6c5"). InnerVolumeSpecName "kube-api-access-t9njz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:27.174785 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.174753 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-envoy" pod "217454ad-7acd-4b25-b91f-885bc464b6c5" (UID: "217454ad-7acd-4b25-b91f-885bc464b6c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:30:27.182789 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.182759 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-token" pod "217454ad-7acd-4b25-b91f-885bc464b6c5" (UID: "217454ad-7acd-4b25-b91f-885bc464b6c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:30:27.187808 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.187779 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-podinfo" pod "217454ad-7acd-4b25-b91f-885bc464b6c5" (UID: "217454ad-7acd-4b25-b91f-885bc464b6c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:30:27.286767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.284774 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:27.286767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.284830 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:27.286767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.284848 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:27.286767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.284865 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:27.286767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.284881 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:27.286767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.284897 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:27.286767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.284915 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9njz\" (UniqueName: \"kubernetes.io/projected/217454ad-7acd-4b25-b91f-885bc464b6c5-kube-api-access-t9njz\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:27.286767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.284930 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/217454ad-7acd-4b25-b91f-885bc464b6c5-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:27.286767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.284943 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/217454ad-7acd-4b25-b91f-885bc464b6c5-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:30:27.920796 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.920769 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" Oct 30 21:30:27.921248 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.920764 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw" event={"ID":"217454ad-7acd-4b25-b91f-885bc464b6c5","Type":"ContainerDied","Data":"e3dd420f98e127cfc3996011e6d8a40644b96a7033470e47d310bf36a6ce75c8"} Oct 30 21:30:27.921248 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.920914 4926 scope.go:122] "RemoveContainer" containerID="301f5a8ab1e97447872c54d0c6049409b4330f445905eb6671ca868a5fec20ad" Oct 30 21:30:27.986404 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:27.986341 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw"] Oct 30 21:30:28.013903 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:28.012830 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-2tsvw"] Oct 30 21:30:29.021880 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:29.021845 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="217454ad-7acd-4b25-b91f-885bc464b6c5" path="/var/lib/kubelet/pods/217454ad-7acd-4b25-b91f-885bc464b6c5/volumes" Oct 30 21:30:29.360258 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:29.360170 4926 patch_prober.go:29] interesting pod/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:30:29.360422 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:29.360268 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" podUID="12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:30:30.968223 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:30.968176 4926 generic.go:356] "Generic (PLEG): container finished" podID="12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" containerID="0a21174f122abf4b93eef30871d437bc2a4f7bf191945c9aaa103461c74a43da" exitCode=0 Oct 30 21:30:30.968785 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:30.968249 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" event={"ID":"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4","Type":"ContainerDied","Data":"0a21174f122abf4b93eef30871d437bc2a4f7bf191945c9aaa103461c74a43da"} Oct 30 21:30:30.968785 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:30.968713 4926 scope.go:122] "RemoveContainer" containerID="0a21174f122abf4b93eef30871d437bc2a4f7bf191945c9aaa103461c74a43da" Oct 30 21:30:30.971005 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:30.970965 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:30:30.971165 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:30.971071 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:30:30.971165 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:30.971120 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:30:31.986054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:31.985571 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" event={"ID":"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4","Type":"ContainerStarted","Data":"c6d339e651788f5bde1866968ea47719d5e5a50fa7d5a2cf955a77a3db54fd5e"} Oct 30 21:30:32.901545 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:32.901514 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:30:32.901892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:32.901871 4926 patch_prober.go:29] interesting pod/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 21:30:32.902074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:32.902050 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" podUID="12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 21:30:33.901845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:33.901809 4926 patch_prober.go:29] interesting pod/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 21:30:33.902662 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:33.902630 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" podUID="12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 21:30:34.900375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:34.900336 4926 patch_prober.go:29] interesting pod/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 21:30:34.900641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:34.900613 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" podUID="12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 21:30:35.918555 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:35.918511 4926 patch_prober.go:29] interesting pod/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 21:30:35.919494 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:35.919456 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" podUID="12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 21:30:36.901391 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:36.901350 4926 patch_prober.go:29] interesting pod/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 30 21:30:36.901597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:36.901424 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" podUID="12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 30 21:30:37.520233 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:30:37.520192 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:30:38.223232 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:38.222994 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:30:38.224745 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:38.224700 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:30:38.225637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:38.225609 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:30:40.514579 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.514536 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2"] Oct 30 21:30:40.516079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516050 4926 cpu_manager.go:414] "containerMap: removing container" podUID="6d846cc6-bbdd-456a-a049-e9d99773172d" containerName="cleanup" Oct 30 21:30:40.516236 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516217 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d846cc6-bbdd-456a-a049-e9d99773172d" containerName="cleanup" Oct 30 21:30:40.516324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516239 4926 cpu_manager.go:414] "containerMap: removing container" podUID="63543c7d-5e17-4248-82f2-b5b947e7aa4c" containerName="cleanup" Oct 30 21:30:40.516324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516248 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63543c7d-5e17-4248-82f2-b5b947e7aa4c" containerName="cleanup" Oct 30 21:30:40.516324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516266 4926 cpu_manager.go:414] "containerMap: removing container" podUID="87424b1f-d6bc-42f6-ad13-b85cc826043f" containerName="istio-proxy" Oct 30 21:30:40.516324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516273 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87424b1f-d6bc-42f6-ad13-b85cc826043f" containerName="istio-proxy" Oct 30 21:30:40.516324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516292 4926 cpu_manager.go:414] "containerMap: removing container" podUID="fce0f54b-5f27-4637-8b5c-d6f2298873bb" containerName="cleanup" Oct 30 21:30:40.516324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516300 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fce0f54b-5f27-4637-8b5c-d6f2298873bb" containerName="cleanup" Oct 30 21:30:40.516324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516313 4926 cpu_manager.go:414] "containerMap: removing container" podUID="810d266f-62a8-4b0f-a146-89ed2e41df99" containerName="cleanup" Oct 30 21:30:40.516324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516320 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="810d266f-62a8-4b0f-a146-89ed2e41df99" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516337 4926 cpu_manager.go:414] "containerMap: removing container" podUID="6d846cc6-bbdd-456a-a049-e9d99773172d" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516344 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d846cc6-bbdd-456a-a049-e9d99773172d" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516361 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c461be4b-e409-48fe-bcaf-6ba8e18ab312" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516369 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c461be4b-e409-48fe-bcaf-6ba8e18ab312" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516382 4926 cpu_manager.go:414] "containerMap: removing container" podUID="731eb7a0-c9e9-41f9-b245-0fe7a00be673" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516389 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="731eb7a0-c9e9-41f9-b245-0fe7a00be673" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516400 4926 cpu_manager.go:414] "containerMap: removing container" podUID="810d266f-62a8-4b0f-a146-89ed2e41df99" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516408 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="810d266f-62a8-4b0f-a146-89ed2e41df99" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516422 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d55b8ddb-6b3e-4f51-8281-f3209c63d949" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516429 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d55b8ddb-6b3e-4f51-8281-f3209c63d949" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516440 4926 cpu_manager.go:414] "containerMap: removing container" podUID="f17fd537-658d-4c4f-9aeb-a763d184c364" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516447 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f17fd537-658d-4c4f-9aeb-a763d184c364" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516460 4926 cpu_manager.go:414] "containerMap: removing container" podUID="495acf58-df75-4721-aaeb-fa3dcb9a3cd9" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516467 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495acf58-df75-4721-aaeb-fa3dcb9a3cd9" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516481 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d922621f-a5d6-4854-8f0e-ba0ee21ade8e" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516489 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d922621f-a5d6-4854-8f0e-ba0ee21ade8e" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516503 4926 cpu_manager.go:414] "containerMap: removing container" podUID="f5d36899-edf3-4bb0-a13b-39c36bf0ed68" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516511 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5d36899-edf3-4bb0-a13b-39c36bf0ed68" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516521 4926 cpu_manager.go:414] "containerMap: removing container" podUID="fce0f54b-5f27-4637-8b5c-d6f2298873bb" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516528 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fce0f54b-5f27-4637-8b5c-d6f2298873bb" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516537 4926 cpu_manager.go:414] "containerMap: removing container" podUID="f5d36899-edf3-4bb0-a13b-39c36bf0ed68" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516544 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5d36899-edf3-4bb0-a13b-39c36bf0ed68" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516558 4926 cpu_manager.go:414] "containerMap: removing container" podUID="0fe4049a-063d-4990-8524-88c95c2d8862" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516566 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe4049a-063d-4990-8524-88c95c2d8862" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516581 4926 cpu_manager.go:414] "containerMap: removing container" podUID="731eb7a0-c9e9-41f9-b245-0fe7a00be673" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516588 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="731eb7a0-c9e9-41f9-b245-0fe7a00be673" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516597 4926 cpu_manager.go:414] "containerMap: removing container" podUID="bde714d2-046b-443a-a5e6-f87c729e5ab0" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516603 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde714d2-046b-443a-a5e6-f87c729e5ab0" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516616 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d55b8ddb-6b3e-4f51-8281-f3209c63d949" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516623 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d55b8ddb-6b3e-4f51-8281-f3209c63d949" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516637 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c461be4b-e409-48fe-bcaf-6ba8e18ab312" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516644 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c461be4b-e409-48fe-bcaf-6ba8e18ab312" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516653 4926 cpu_manager.go:414] "containerMap: removing container" podUID="f17fd537-658d-4c4f-9aeb-a763d184c364" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516660 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f17fd537-658d-4c4f-9aeb-a763d184c364" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516676 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d922621f-a5d6-4854-8f0e-ba0ee21ade8e" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516683 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d922621f-a5d6-4854-8f0e-ba0ee21ade8e" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516704 4926 cpu_manager.go:414] "containerMap: removing container" podUID="f17fd537-658d-4c4f-9aeb-a763d184c364" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516712 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f17fd537-658d-4c4f-9aeb-a763d184c364" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516730 4926 cpu_manager.go:414] "containerMap: removing container" podUID="fd7f7c91-0516-476c-afc8-94130b2e3439" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516737 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd7f7c91-0516-476c-afc8-94130b2e3439" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516748 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d922621f-a5d6-4854-8f0e-ba0ee21ade8e" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516754 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d922621f-a5d6-4854-8f0e-ba0ee21ade8e" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516766 4926 cpu_manager.go:414] "containerMap: removing container" podUID="63543c7d-5e17-4248-82f2-b5b947e7aa4c" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516772 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63543c7d-5e17-4248-82f2-b5b947e7aa4c" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516780 4926 cpu_manager.go:414] "containerMap: removing container" podUID="63543c7d-5e17-4248-82f2-b5b947e7aa4c" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516787 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63543c7d-5e17-4248-82f2-b5b947e7aa4c" containerName="cleanup" Oct 30 21:30:40.516752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516797 4926 cpu_manager.go:414] "containerMap: removing container" podUID="f5d36899-edf3-4bb0-a13b-39c36bf0ed68" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516804 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5d36899-edf3-4bb0-a13b-39c36bf0ed68" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516821 4926 cpu_manager.go:414] "containerMap: removing container" podUID="495acf58-df75-4721-aaeb-fa3dcb9a3cd9" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516828 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495acf58-df75-4721-aaeb-fa3dcb9a3cd9" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516841 4926 cpu_manager.go:414] "containerMap: removing container" podUID="0fe4049a-063d-4990-8524-88c95c2d8862" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516847 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe4049a-063d-4990-8524-88c95c2d8862" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516869 4926 cpu_manager.go:414] "containerMap: removing container" podUID="495acf58-df75-4721-aaeb-fa3dcb9a3cd9" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516875 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495acf58-df75-4721-aaeb-fa3dcb9a3cd9" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516886 4926 cpu_manager.go:414] "containerMap: removing container" podUID="0fe4049a-063d-4990-8524-88c95c2d8862" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516893 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe4049a-063d-4990-8524-88c95c2d8862" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516914 4926 cpu_manager.go:414] "containerMap: removing container" podUID="bde714d2-046b-443a-a5e6-f87c729e5ab0" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516920 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde714d2-046b-443a-a5e6-f87c729e5ab0" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516932 4926 cpu_manager.go:414] "containerMap: removing container" podUID="fd7f7c91-0516-476c-afc8-94130b2e3439" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516940 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd7f7c91-0516-476c-afc8-94130b2e3439" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516949 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d55b8ddb-6b3e-4f51-8281-f3209c63d949" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516955 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d55b8ddb-6b3e-4f51-8281-f3209c63d949" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516967 4926 cpu_manager.go:414] "containerMap: removing container" podUID="bde714d2-046b-443a-a5e6-f87c729e5ab0" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516973 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bde714d2-046b-443a-a5e6-f87c729e5ab0" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516982 4926 cpu_manager.go:414] "containerMap: removing container" podUID="fd7f7c91-0516-476c-afc8-94130b2e3439" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.516990 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd7f7c91-0516-476c-afc8-94130b2e3439" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517005 4926 cpu_manager.go:414] "containerMap: removing container" podUID="217454ad-7acd-4b25-b91f-885bc464b6c5" containerName="istio-proxy" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517012 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="217454ad-7acd-4b25-b91f-885bc464b6c5" containerName="istio-proxy" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517219 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="495acf58-df75-4721-aaeb-fa3dcb9a3cd9" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517235 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe4049a-063d-4990-8524-88c95c2d8862" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517250 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde714d2-046b-443a-a5e6-f87c729e5ab0" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517260 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd7f7c91-0516-476c-afc8-94130b2e3439" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517270 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="fce0f54b-5f27-4637-8b5c-d6f2298873bb" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517279 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d55b8ddb-6b3e-4f51-8281-f3209c63d949" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517287 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="731eb7a0-c9e9-41f9-b245-0fe7a00be673" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517294 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde714d2-046b-443a-a5e6-f87c729e5ab0" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517302 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd7f7c91-0516-476c-afc8-94130b2e3439" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517311 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="217454ad-7acd-4b25-b91f-885bc464b6c5" containerName="istio-proxy" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517322 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c461be4b-e409-48fe-bcaf-6ba8e18ab312" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517330 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d846cc6-bbdd-456a-a049-e9d99773172d" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517339 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="63543c7d-5e17-4248-82f2-b5b947e7aa4c" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517348 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="87424b1f-d6bc-42f6-ad13-b85cc826043f" containerName="istio-proxy" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517359 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="fce0f54b-5f27-4637-8b5c-d6f2298873bb" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517369 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="810d266f-62a8-4b0f-a146-89ed2e41df99" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517377 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d846cc6-bbdd-456a-a049-e9d99773172d" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517388 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c461be4b-e409-48fe-bcaf-6ba8e18ab312" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517397 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d846cc6-bbdd-456a-a049-e9d99773172d" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517408 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="f17fd537-658d-4c4f-9aeb-a763d184c364" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517418 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="495acf58-df75-4721-aaeb-fa3dcb9a3cd9" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517428 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d922621f-a5d6-4854-8f0e-ba0ee21ade8e" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517436 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5d36899-edf3-4bb0-a13b-39c36bf0ed68" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517444 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5d36899-edf3-4bb0-a13b-39c36bf0ed68" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517453 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="731eb7a0-c9e9-41f9-b245-0fe7a00be673" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517462 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d55b8ddb-6b3e-4f51-8281-f3209c63d949" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517471 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c461be4b-e409-48fe-bcaf-6ba8e18ab312" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517478 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="f17fd537-658d-4c4f-9aeb-a763d184c364" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517490 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="f17fd537-658d-4c4f-9aeb-a763d184c364" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517499 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="810d266f-62a8-4b0f-a146-89ed2e41df99" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517509 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d922621f-a5d6-4854-8f0e-ba0ee21ade8e" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517517 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="63543c7d-5e17-4248-82f2-b5b947e7aa4c" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517526 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="63543c7d-5e17-4248-82f2-b5b947e7aa4c" containerName="cleanup" Oct 30 21:30:40.519157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.517536 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe4049a-063d-4990-8524-88c95c2d8862" containerName="cleanup" Oct 30 21:30:40.522334 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.522260 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:40.542266 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.542233 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2"] Oct 30 21:30:40.544209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.544123 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c5145c\"/\"test-model-c5145c-kserve-self-signed-certs\"" Oct 30 21:30:40.544209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.544124 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c5145c\"/\"openshift-service-ca.crt\"" Oct 30 21:30:40.544209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.544169 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c5145c\"/\"default-dockercfg-rmcq2\"" Oct 30 21:30:40.544209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.544187 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c5145c\"/\"kube-root-ca.crt\"" Oct 30 21:30:40.569976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.569016 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-dshm\") pod \"test-model-c5145c-kserve-5fd549-697m2\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:40.569976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.569101 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-tmp-dir\") pod \"test-model-c5145c-kserve-5fd549-697m2\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:40.569976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.569143 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-model-cache\") pod \"test-model-c5145c-kserve-5fd549-697m2\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:40.569976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.569202 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-tls-certs\") pod \"test-model-c5145c-kserve-5fd549-697m2\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:40.569976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.569328 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr6bb\" (UniqueName: \"kubernetes.io/projected/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-kube-api-access-kr6bb\") pod \"test-model-c5145c-kserve-5fd549-697m2\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:40.569976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.569372 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-home\") pod \"test-model-c5145c-kserve-5fd549-697m2\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:40.671567 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.671517 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-dshm\") pod \"test-model-c5145c-kserve-5fd549-697m2\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:40.671771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.671583 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-tmp-dir\") pod \"test-model-c5145c-kserve-5fd549-697m2\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:40.671771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.671643 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-model-cache\") pod \"test-model-c5145c-kserve-5fd549-697m2\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:40.671771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.671676 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-tls-certs\") pod \"test-model-c5145c-kserve-5fd549-697m2\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:40.671950 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.671847 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6bb\" (UniqueName: \"kubernetes.io/projected/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-kube-api-access-kr6bb\") pod \"test-model-c5145c-kserve-5fd549-697m2\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:40.671950 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.671923 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-home\") pod \"test-model-c5145c-kserve-5fd549-697m2\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:40.672089 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.672020 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-tmp-dir\") pod \"test-model-c5145c-kserve-5fd549-697m2\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:40.672546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.672397 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-model-cache\") pod \"test-model-c5145c-kserve-5fd549-697m2\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:40.672546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.672511 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-home\") pod \"test-model-c5145c-kserve-5fd549-697m2\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:40.674857 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.674831 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-dshm\") pod \"test-model-c5145c-kserve-5fd549-697m2\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:40.675314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.675294 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-tls-certs\") pod \"test-model-c5145c-kserve-5fd549-697m2\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:40.692187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.692159 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr6bb\" (UniqueName: \"kubernetes.io/projected/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-kube-api-access-kr6bb\") pod \"test-model-c5145c-kserve-5fd549-697m2\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:40.856120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:40.855368 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:41.385043 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.384995 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v"] Oct 30 21:30:41.387187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.386056 4926 cpu_manager.go:414] "containerMap: removing container" podUID="810d266f-62a8-4b0f-a146-89ed2e41df99" containerName="cleanup" Oct 30 21:30:41.387187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.386090 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="810d266f-62a8-4b0f-a146-89ed2e41df99" containerName="cleanup" Oct 30 21:30:41.387187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.386123 4926 cpu_manager.go:414] "containerMap: removing container" podUID="731eb7a0-c9e9-41f9-b245-0fe7a00be673" containerName="cleanup" Oct 30 21:30:41.387187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.386131 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="731eb7a0-c9e9-41f9-b245-0fe7a00be673" containerName="cleanup" Oct 30 21:30:41.387187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.386146 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c461be4b-e409-48fe-bcaf-6ba8e18ab312" containerName="cleanup" Oct 30 21:30:41.387187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.386154 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c461be4b-e409-48fe-bcaf-6ba8e18ab312" containerName="cleanup" Oct 30 21:30:41.387187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.386219 4926 cpu_manager.go:414] "containerMap: removing container" podUID="fce0f54b-5f27-4637-8b5c-d6f2298873bb" containerName="cleanup" Oct 30 21:30:41.387187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.386227 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fce0f54b-5f27-4637-8b5c-d6f2298873bb" containerName="cleanup" Oct 30 21:30:41.387187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.386242 4926 cpu_manager.go:414] "containerMap: removing container" podUID="6d846cc6-bbdd-456a-a049-e9d99773172d" containerName="cleanup" Oct 30 21:30:41.387187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.386249 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d846cc6-bbdd-456a-a049-e9d99773172d" containerName="cleanup" Oct 30 21:30:41.387187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.386390 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="731eb7a0-c9e9-41f9-b245-0fe7a00be673" containerName="cleanup" Oct 30 21:30:41.387187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.386410 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="495acf58-df75-4721-aaeb-fa3dcb9a3cd9" containerName="cleanup" Oct 30 21:30:41.387187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.386422 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe4049a-063d-4990-8524-88c95c2d8862" containerName="cleanup" Oct 30 21:30:41.387187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.386435 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d55b8ddb-6b3e-4f51-8281-f3209c63d949" containerName="cleanup" Oct 30 21:30:41.387187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.386447 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="bde714d2-046b-443a-a5e6-f87c729e5ab0" containerName="cleanup" Oct 30 21:30:41.387187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.386458 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd7f7c91-0516-476c-afc8-94130b2e3439" containerName="cleanup" Oct 30 21:30:41.387187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.386471 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="fce0f54b-5f27-4637-8b5c-d6f2298873bb" containerName="cleanup" Oct 30 21:30:41.387187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.386481 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="810d266f-62a8-4b0f-a146-89ed2e41df99" containerName="cleanup" Oct 30 21:30:41.387187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.386495 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d922621f-a5d6-4854-8f0e-ba0ee21ade8e" containerName="cleanup" Oct 30 21:30:41.387187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.386504 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5d36899-edf3-4bb0-a13b-39c36bf0ed68" containerName="cleanup" Oct 30 21:30:41.398763 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.392555 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v"] Oct 30 21:30:41.399076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.399056 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.430183 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.430012 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3b28bc-openshift-default-dockercfg-64znn\"" Oct 30 21:30:41.499922 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.499896 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2"] Oct 30 21:30:41.501391 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:30:41.501366 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ae29b4c_757d_49a0_9dd0_fa0999ab01c4.slice/crio-0de5a0fcd025f10d5242fde4f2a3e1206f3f82e550ae2dd8f15bed7183dbf848 WatchSource:0}: Error finding container 0de5a0fcd025f10d5242fde4f2a3e1206f3f82e550ae2dd8f15bed7183dbf848: Status 404 returned error can't find the container with id 0de5a0fcd025f10d5242fde4f2a3e1206f3f82e550ae2dd8f15bed7183dbf848 Oct 30 21:30:41.533993 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.533968 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-podinfo\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.534370 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.534012 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-workload-socket\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.534370 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.534173 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-token\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.534370 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.534241 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c795d0-9420-4f20-862a-7c09c66bd016-istiod-ca-cert\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.534370 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.534274 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfdvz\" (UniqueName: \"kubernetes.io/projected/f0c795d0-9420-4f20-862a-7c09c66bd016-kube-api-access-cfdvz\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.534553 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.534388 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-envoy\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.534553 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.534494 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-workload-certs\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.534553 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.534516 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-credential-socket\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.534668 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.534595 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-data\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.635680 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.635611 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-data\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.635819 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.635774 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-podinfo\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.635819 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.635799 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-workload-socket\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.635898 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.635863 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-token\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.635945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.635902 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c795d0-9420-4f20-862a-7c09c66bd016-istiod-ca-cert\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.635945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.635927 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfdvz\" (UniqueName: \"kubernetes.io/projected/f0c795d0-9420-4f20-862a-7c09c66bd016-kube-api-access-cfdvz\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.636065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.635981 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-data\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.636065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.636007 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-envoy\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.636261 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.636141 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-workload-certs\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.636261 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.636183 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-credential-socket\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.636478 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.636458 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-workload-socket\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.636580 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.636533 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-workload-certs\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.636667 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.636646 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-credential-socket\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.636733 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.636706 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c795d0-9420-4f20-862a-7c09c66bd016-istiod-ca-cert\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.638323 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.638301 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-envoy\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.638539 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.638522 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-podinfo\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.654099 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.654068 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-token\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.654206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.654186 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfdvz\" (UniqueName: \"kubernetes.io/projected/f0c795d0-9420-4f20-862a-7c09c66bd016-kube-api-access-cfdvz\") pod \"e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:41.743348 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:41.743316 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:42.069632 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:42.069601 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v"] Oct 30 21:30:42.069902 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:30:42.069874 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c795d0_9420_4f20_862a_7c09c66bd016.slice/crio-077b027abbec215331e7b32924f90328744bcc4e133abb130021bc330fa4c96b WatchSource:0}: Error finding container 077b027abbec215331e7b32924f90328744bcc4e133abb130021bc330fa4c96b: Status 404 returned error can't find the container with id 077b027abbec215331e7b32924f90328744bcc4e133abb130021bc330fa4c96b Oct 30 21:30:42.073778 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:42.073726 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:30:42.073885 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:42.073823 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:30:42.073885 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:42.073865 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:30:42.415593 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:42.415554 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" event={"ID":"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4","Type":"ContainerStarted","Data":"5c39031f7254c4d641e8fbf20d67f61a6385fb260eea6a61c831f87b9cd521d6"} Oct 30 21:30:42.419165 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:42.419130 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" event={"ID":"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4","Type":"ContainerStarted","Data":"0de5a0fcd025f10d5242fde4f2a3e1206f3f82e550ae2dd8f15bed7183dbf848"} Oct 30 21:30:42.420326 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:42.420295 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" event={"ID":"f0c795d0-9420-4f20-862a-7c09c66bd016","Type":"ContainerStarted","Data":"40ec18a10f41377c268c1b1d7590119ad4063ceea6d74fdd9ec82f5a5db5a2f9"} Oct 30 21:30:42.420458 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:42.420395 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" event={"ID":"f0c795d0-9420-4f20-862a-7c09c66bd016","Type":"ContainerStarted","Data":"077b027abbec215331e7b32924f90328744bcc4e133abb130021bc330fa4c96b"} Oct 30 21:30:42.468807 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:42.468749 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" podStartSLOduration=2.468735729 podStartE2EDuration="2.468735729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:42.464595216 +0000 UTC m=+1829.585668965" watchObservedRunningTime="2026-10-30 21:30:42.468735729 +0000 UTC m=+1829.589809474" Oct 30 21:30:42.509109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:42.508991 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" podStartSLOduration=1.508978209 podStartE2EDuration="1.508978209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:42.50370169 +0000 UTC m=+1829.624775438" watchObservedRunningTime="2026-10-30 21:30:42.508978209 +0000 UTC m=+1829.630051955" Oct 30 21:30:42.743774 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:42.743682 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:42.855825 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:42.855793 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:43.746167 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:43.746127 4926 patch_prober.go:29] interesting pod/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 21:30:43.746610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:43.746185 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" podUID="f0c795d0-9420-4f20-862a-7c09c66bd016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 21:30:44.747055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:44.746869 4926 patch_prober.go:29] interesting pod/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 21:30:44.747055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:44.746935 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" podUID="f0c795d0-9420-4f20-862a-7c09c66bd016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 21:30:45.743595 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:45.743562 4926 patch_prober.go:29] interesting pod/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 21:30:45.743784 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:45.743616 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" podUID="f0c795d0-9420-4f20-862a-7c09c66bd016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 21:30:46.744120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:46.744075 4926 patch_prober.go:29] interesting pod/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 21:30:46.744544 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:46.744150 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" podUID="f0c795d0-9420-4f20-862a-7c09c66bd016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 21:30:47.743951 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:47.743909 4926 patch_prober.go:29] interesting pod/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 21:30:47.744147 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:47.743980 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" podUID="f0c795d0-9420-4f20-862a-7c09c66bd016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 21:30:48.304911 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.304860 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn"] Oct 30 21:30:48.316101 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.314264 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.324050 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.321645 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn"] Oct 30 21:30:48.333537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.333068 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-credential-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.333537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.333117 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-podinfo\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.333537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.333148 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-envoy\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.333537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.333218 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-workload-certs\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.333537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.333311 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-data\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.333537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.333338 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b985e6-7718-4c16-896b-960d0f1d744e-istiod-ca-cert\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.333537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.333414 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bw7c\" (UniqueName: \"kubernetes.io/projected/c3b985e6-7718-4c16-896b-960d0f1d744e-kube-api-access-6bw7c\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.333537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.333491 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-token\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.333537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.333522 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-workload-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.438300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.435662 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-credential-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.438300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.435731 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-podinfo\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.438300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.435760 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-envoy\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.438300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.435851 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-workload-certs\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.438300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.435976 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-data\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.438300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.436006 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b985e6-7718-4c16-896b-960d0f1d744e-istiod-ca-cert\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.438300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.436124 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bw7c\" (UniqueName: \"kubernetes.io/projected/c3b985e6-7718-4c16-896b-960d0f1d744e-kube-api-access-6bw7c\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.438300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.436208 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-credential-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.438300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.436214 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-token\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.438300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.436540 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-workload-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.438300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.436990 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-workload-socket\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.438300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.437264 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-data\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.438300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.437804 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-workload-certs\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.438300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.437912 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b985e6-7718-4c16-896b-960d0f1d744e-istiod-ca-cert\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.439749 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.439721 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-podinfo\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.441470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.441449 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-envoy\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.455604 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.455578 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-token\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.456204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.456184 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bw7c\" (UniqueName: \"kubernetes.io/projected/c3b985e6-7718-4c16-896b-960d0f1d744e-kube-api-access-6bw7c\") pod \"e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.640153 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.640060 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:48.744397 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.744361 4926 patch_prober.go:29] interesting pod/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 21:30:48.744738 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:48.744426 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" podUID="f0c795d0-9420-4f20-862a-7c09c66bd016" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 21:30:49.052218 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:49.050888 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn"] Oct 30 21:30:49.052218 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:30:49.051542 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b985e6_7718_4c16_896b_960d0f1d744e.slice/crio-0b71674d0e50b9f30474e18cacec9d7c7a47f0d9aae4d6f1542240ed4e822c91 WatchSource:0}: Error finding container 0b71674d0e50b9f30474e18cacec9d7c7a47f0d9aae4d6f1542240ed4e822c91: Status 404 returned error can't find the container with id 0b71674d0e50b9f30474e18cacec9d7c7a47f0d9aae4d6f1542240ed4e822c91 Oct 30 21:30:49.054734 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:49.054694 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:30:49.054840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:49.054786 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:30:49.054840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:49.054833 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:30:49.566180 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:49.566140 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" event={"ID":"c3b985e6-7718-4c16-896b-960d0f1d744e","Type":"ContainerStarted","Data":"a54b81410563246f57638c188c3b09e679129371df96ee608e58b77f451b885c"} Oct 30 21:30:49.566399 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:49.566208 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" event={"ID":"c3b985e6-7718-4c16-896b-960d0f1d744e","Type":"ContainerStarted","Data":"0b71674d0e50b9f30474e18cacec9d7c7a47f0d9aae4d6f1542240ed4e822c91"} Oct 30 21:30:49.641320 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:49.641288 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:49.642826 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:49.642766 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" podStartSLOduration=1.6427487109999999 podStartE2EDuration="1.642748711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:30:49.625717023 +0000 UTC m=+1836.746790776" watchObservedRunningTime="2026-10-30 21:30:49.642748711 +0000 UTC m=+1836.763822471" Oct 30 21:30:49.758058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:49.757551 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:49.758515 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:49.758302 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:49.759335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:49.759312 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:30:50.642792 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:50.642506 4926 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 21:30:50.642792 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:50.642551 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" podUID="c3b985e6-7718-4c16-896b-960d0f1d744e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 21:30:50.874451 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:50.871340 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:51.242691 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:51.242653 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:51.633677 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:51.633605 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:30:51.641561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:51.641536 4926 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 21:30:51.641695 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:51.641584 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" podUID="c3b985e6-7718-4c16-896b-960d0f1d744e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 21:30:52.641645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:52.641612 4926 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 21:30:52.642104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:52.641660 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" podUID="c3b985e6-7718-4c16-896b-960d0f1d744e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 21:30:53.641143 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:53.641102 4926 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 21:30:53.641330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:53.641172 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" podUID="c3b985e6-7718-4c16-896b-960d0f1d744e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 21:30:54.641740 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:54.641704 4926 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 21:30:54.642254 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:54.641760 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" podUID="c3b985e6-7718-4c16-896b-960d0f1d744e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 21:30:55.646489 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:55.646454 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:55.647012 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:55.646991 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:30:55.647758 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:30:55.647738 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:31:07.490051 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:31:07.489994 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:31:15.530571 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:15.530533 4926 scope.go:122] "RemoveContainer" containerID="2d8ac12ee91f4ba33bb77b725247c0959814c9bb53cf580bab59c177a45ace35" Oct 30 21:31:15.542401 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:15.542378 4926 scope.go:122] "RemoveContainer" containerID="18d4353b7e18462cb185923e83a06f5095ef74ce0abd04576aabee15e0d552d1" Oct 30 21:31:15.553894 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:15.553872 4926 scope.go:122] "RemoveContainer" containerID="d0e69f971a6efb7df3561d55caed05dc887f59890bc8399b3991f3d6ece1352e" Oct 30 21:31:21.515198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.515158 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh"] Oct 30 21:31:21.527969 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.523108 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:21.527969 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.527882 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh"] Oct 30 21:31:21.549086 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.549016 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-abf04b\"/\"test-model-abf04b-kserve-self-signed-certs\"" Oct 30 21:31:21.549086 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.549019 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-abf04b\"/\"openshift-service-ca.crt\"" Oct 30 21:31:21.549316 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.549104 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-abf04b\"/\"default-dockercfg-v78w6\"" Oct 30 21:31:21.549316 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.549122 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-abf04b\"/\"kube-root-ca.crt\"" Oct 30 21:31:21.556081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.551360 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-dshm\") pod \"test-model-abf04b-kserve-c7fcd8789-l7ngh\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:21.562388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.556309 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87zd8\" (UniqueName: \"kubernetes.io/projected/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-kube-api-access-87zd8\") pod \"test-model-abf04b-kserve-c7fcd8789-l7ngh\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:21.562388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.556367 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-model-cache\") pod \"test-model-abf04b-kserve-c7fcd8789-l7ngh\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:21.562388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.556454 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-tmp-dir\") pod \"test-model-abf04b-kserve-c7fcd8789-l7ngh\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:21.562388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.556613 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-home\") pod \"test-model-abf04b-kserve-c7fcd8789-l7ngh\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:21.562388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.556708 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-tls-certs\") pod \"test-model-abf04b-kserve-c7fcd8789-l7ngh\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:21.658749 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.658710 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-dshm\") pod \"test-model-abf04b-kserve-c7fcd8789-l7ngh\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:21.658935 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.658782 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87zd8\" (UniqueName: \"kubernetes.io/projected/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-kube-api-access-87zd8\") pod \"test-model-abf04b-kserve-c7fcd8789-l7ngh\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:21.658935 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.658816 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-model-cache\") pod \"test-model-abf04b-kserve-c7fcd8789-l7ngh\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:21.658935 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.658895 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-tmp-dir\") pod \"test-model-abf04b-kserve-c7fcd8789-l7ngh\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:21.659159 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.659064 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-home\") pod \"test-model-abf04b-kserve-c7fcd8789-l7ngh\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:21.659159 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.659147 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-tls-certs\") pod \"test-model-abf04b-kserve-c7fcd8789-l7ngh\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:21.659371 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.659342 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-model-cache\") pod \"test-model-abf04b-kserve-c7fcd8789-l7ngh\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:21.659493 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.659466 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-home\") pod \"test-model-abf04b-kserve-c7fcd8789-l7ngh\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:21.659735 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.659712 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-tmp-dir\") pod \"test-model-abf04b-kserve-c7fcd8789-l7ngh\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:21.661307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.661283 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-dshm\") pod \"test-model-abf04b-kserve-c7fcd8789-l7ngh\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:21.662386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.662365 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-tls-certs\") pod \"test-model-abf04b-kserve-c7fcd8789-l7ngh\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:21.677533 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.677500 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87zd8\" (UniqueName: \"kubernetes.io/projected/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-kube-api-access-87zd8\") pod \"test-model-abf04b-kserve-c7fcd8789-l7ngh\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:21.869110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:21.867316 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:22.100091 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:22.099496 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:31:22.300507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:22.287563 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh"] Oct 30 21:31:22.300507 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:31:22.289134 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaac5cd64_6203_43ee_aa05_d5b8c1e0240a.slice/crio-b043ed3e47497f0ed58dbbd9c8bcdf2ba692834005232d3961eb439325df18c2 WatchSource:0}: Error finding container b043ed3e47497f0ed58dbbd9c8bcdf2ba692834005232d3961eb439325df18c2: Status 404 returned error can't find the container with id b043ed3e47497f0ed58dbbd9c8bcdf2ba692834005232d3961eb439325df18c2 Oct 30 21:31:22.985651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:22.985613 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" event={"ID":"aac5cd64-6203-43ee-aa05-d5b8c1e0240a","Type":"ContainerStarted","Data":"5e201a253f5363bfedf22643a9fed3a1fe43e5a4458d9036928bea8ea4b902a8"} Oct 30 21:31:22.985651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:22.985663 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" event={"ID":"aac5cd64-6203-43ee-aa05-d5b8c1e0240a","Type":"ContainerStarted","Data":"b043ed3e47497f0ed58dbbd9c8bcdf2ba692834005232d3961eb439325df18c2"} Oct 30 21:31:23.028119 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:23.028049 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" podStartSLOduration=2.028020742 podStartE2EDuration="2.028020742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:31:23.02387621 +0000 UTC m=+1870.144949955" watchObservedRunningTime="2026-10-30 21:31:23.028020742 +0000 UTC m=+1870.149094532" Oct 30 21:31:23.868779 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:23.868750 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:31.868554 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:31.868513 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:31.882105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:31.882071 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:32.105138 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:32.105107 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:31:34.115822 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:34.115739 4926 generic.go:356] "Generic (PLEG): container finished" podID="ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" containerID="62d4396fc9cec9e296628ccfc5a668a4237aea8fab4eacf7bc264d64fe6d9a4e" exitCode=0 Oct 30 21:31:34.115822 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:34.115801 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" event={"ID":"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d","Type":"ContainerDied","Data":"62d4396fc9cec9e296628ccfc5a668a4237aea8fab4eacf7bc264d64fe6d9a4e"} Oct 30 21:31:34.116328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:34.116312 4926 scope.go:122] "RemoveContainer" containerID="62d4396fc9cec9e296628ccfc5a668a4237aea8fab4eacf7bc264d64fe6d9a4e" Oct 30 21:31:34.118413 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:34.118387 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:31:34.118493 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:34.118454 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:31:34.118493 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:34.118487 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:31:34.839054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:34.831824 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm"] Oct 30 21:31:34.839054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:34.832236 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" podUID="c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" containerName="istio-proxy" containerID="cri-o://c9ae83175007b668ac93f22fb889cad519f1241de85540d01101e5a57709ccff" gracePeriod=30 Oct 30 21:31:35.125986 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:35.125909 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" event={"ID":"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d","Type":"ContainerStarted","Data":"e4c60155bf64038f95ee7b41ab80fca992205d1ffc68b88115c5d56f12e0de88"} Oct 30 21:31:35.796057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:35.786438 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:31:35.796057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:35.786542 4926 patch_prober.go:29] interesting pod/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 21:31:35.796057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:35.786582 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" podUID="ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 21:31:36.787055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:36.786487 4926 patch_prober.go:29] interesting pod/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 21:31:36.787055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:36.786566 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" podUID="ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 21:31:37.509500 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:31:37.509463 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:31:37.791307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:37.791196 4926 patch_prober.go:29] interesting pod/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 21:31:37.791307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:37.791258 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" podUID="ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 21:31:38.034982 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:38.034939 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cee0a8d7-qbghx"] Oct 30 21:31:38.043070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:38.042973 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cee0a8d7-qbghx"] Oct 30 21:31:38.788055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:38.787416 4926 patch_prober.go:29] interesting pod/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 21:31:38.788055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:38.787483 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" podUID="ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 21:31:39.017708 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:39.017674 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a804628a-e494-4335-9871-5e317b38c9dd" path="/var/lib/kubelet/pods/a804628a-e494-4335-9871-5e317b38c9dd/volumes" Oct 30 21:31:39.785481 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:39.785442 4926 patch_prober.go:29] interesting pod/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Oct 30 21:31:39.786336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:39.786297 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" podUID="ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Oct 30 21:31:40.192055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.191382 4926 patch_prober.go:29] interesting pod/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 30 21:31:40.192055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.191459 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" podUID="c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 30 21:31:40.197420 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.197384 4926 generic.go:356] "Generic (PLEG): container finished" podID="c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" containerID="c9ae83175007b668ac93f22fb889cad519f1241de85540d01101e5a57709ccff" exitCode=0 Oct 30 21:31:40.197594 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.197577 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" event={"ID":"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7","Type":"ContainerDied","Data":"c9ae83175007b668ac93f22fb889cad519f1241de85540d01101e5a57709ccff"} Oct 30 21:31:40.596138 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.596108 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:31:40.609530 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.609499 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-podinfo\") pod \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " Oct 30 21:31:40.609700 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.609594 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-envoy\") pod \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " Oct 30 21:31:40.609764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.609743 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istiod-ca-cert\") pod \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " Oct 30 21:31:40.609826 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.609791 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-workload-certs\") pod \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " Oct 30 21:31:40.609887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.609830 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-data\") pod \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " Oct 30 21:31:40.609942 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.609899 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-credential-socket\") pod \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " Oct 30 21:31:40.609996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.609942 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-kube-api-access-92lhn\" (UniqueName: \"kubernetes.io/projected/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-kube-api-access-92lhn\") pod \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " Oct 30 21:31:40.610070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.610011 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-workload-socket\") pod \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " Oct 30 21:31:40.610133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.610118 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-token\" (UniqueName: \"kubernetes.io/projected/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-token\") pod \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\" (UID: \"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7\") " Oct 30 21:31:40.610363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.610300 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-credential-socket" pod "c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" (UID: "c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:40.610872 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.610844 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:31:40.611188 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.611166 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istiod-ca-cert" pod "c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" (UID: "c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:31:40.611708 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.611682 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-data" pod "c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" (UID: "c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:40.613716 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.611921 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-workload-certs" pod "c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" (UID: "c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:40.613716 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.612096 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-workload-socket" pod "c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" (UID: "c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:40.613716 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.612233 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-podinfo" pod "c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" (UID: "c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:31:40.613716 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.613252 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-envoy" pod "c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" (UID: "c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:31:40.613991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.613970 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-token" pod "c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" (UID: "c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:40.614157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.614135 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-kube-api-access-92lhn" pod "c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" (UID: "c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7"). InnerVolumeSpecName "kube-api-access-92lhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:31:40.712856 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.712765 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:31:40.712856 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.712804 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:31:40.712856 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.712820 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92lhn\" (UniqueName: \"kubernetes.io/projected/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-kube-api-access-92lhn\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:31:40.712856 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.712834 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:31:40.712856 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.712848 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:31:40.712856 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.712861 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:31:40.713388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.712874 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:31:40.713388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.712886 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:31:40.989136 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.988987 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:31:40.991137 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:40.991081 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:31:41.008817 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:41.008299 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:31:41.208526 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:41.208497 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" Oct 30 21:31:41.208526 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:41.208518 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm" event={"ID":"c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7","Type":"ContainerDied","Data":"e54255a2196dea3a432a1a19bfc59a3766a2d18610b9e4c69050944b5347e64b"} Oct 30 21:31:41.209107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:41.208584 4926 scope.go:122] "RemoveContainer" containerID="c9ae83175007b668ac93f22fb889cad519f1241de85540d01101e5a57709ccff" Oct 30 21:31:41.264067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:41.263976 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm"] Oct 30 21:31:41.274387 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:41.274356 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ed088d-openshift-default-6d4f744f5-ncgmm"] Oct 30 21:31:43.019176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:31:43.019142 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" path="/var/lib/kubelet/pods/c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7/volumes" Oct 30 21:32:07.493916 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:32:07.493873 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:32:15.723281 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:15.723248 4926 scope.go:122] "RemoveContainer" containerID="a9da208783ebd08d7165b2bb7331a6902341908d170ab9904406a19b65fcdd2f" Oct 30 21:32:19.430985 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:19.430942 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc"] Oct 30 21:32:19.431536 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:19.431336 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" podUID="b4688f01-eef1-4e8f-9370-05fbecbd6f29" containerName="istio-proxy" containerID="cri-o://6d1357db6dfd71442df6f57b27ee4b780b554ae4a10551396661ffac4691514e" gracePeriod=30 Oct 30 21:32:23.997434 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:23.997390 4926 patch_prober.go:29] interesting pod/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:32:23.997981 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:23.997474 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" podUID="b4688f01-eef1-4e8f-9370-05fbecbd6f29" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:32:24.731579 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.731486 4926 generic.go:356] "Generic (PLEG): container finished" podID="b4688f01-eef1-4e8f-9370-05fbecbd6f29" containerID="6d1357db6dfd71442df6f57b27ee4b780b554ae4a10551396661ffac4691514e" exitCode=0 Oct 30 21:32:24.731810 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.731786 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" event={"ID":"b4688f01-eef1-4e8f-9370-05fbecbd6f29","Type":"ContainerDied","Data":"6d1357db6dfd71442df6f57b27ee4b780b554ae4a10551396661ffac4691514e"} Oct 30 21:32:24.794687 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.794658 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:32:24.904879 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.904792 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-envoy\") pod \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " Oct 30 21:32:24.904879 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.904847 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istiod-ca-cert\") pod \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " Oct 30 21:32:24.905152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.904914 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-workload-certs\") pod \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " Oct 30 21:32:24.905152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.905087 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-token\" (UniqueName: \"kubernetes.io/projected/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-token\") pod \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " Oct 30 21:32:24.905269 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.905149 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-podinfo\") pod \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " Oct 30 21:32:24.905269 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.905213 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-data\") pod \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " Oct 30 21:32:24.905378 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.905318 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-credential-socket\") pod \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " Oct 30 21:32:24.905511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.905494 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-workload-socket\") pod \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " Oct 30 21:32:24.905571 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.905546 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4688f01-eef1-4e8f-9370-05fbecbd6f29-kube-api-access-9l858\" (UniqueName: \"kubernetes.io/projected/b4688f01-eef1-4e8f-9370-05fbecbd6f29-kube-api-access-9l858\") pod \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\" (UID: \"b4688f01-eef1-4e8f-9370-05fbecbd6f29\") " Oct 30 21:32:24.906799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.906570 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istiod-ca-cert" pod "b4688f01-eef1-4e8f-9370-05fbecbd6f29" (UID: "b4688f01-eef1-4e8f-9370-05fbecbd6f29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:24.906923 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.906828 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-workload-socket" pod "b4688f01-eef1-4e8f-9370-05fbecbd6f29" (UID: "b4688f01-eef1-4e8f-9370-05fbecbd6f29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:24.906923 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.906902 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-workload-certs" pod "b4688f01-eef1-4e8f-9370-05fbecbd6f29" (UID: "b4688f01-eef1-4e8f-9370-05fbecbd6f29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:24.907322 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.907288 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-credential-socket" pod "b4688f01-eef1-4e8f-9370-05fbecbd6f29" (UID: "b4688f01-eef1-4e8f-9370-05fbecbd6f29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:24.907621 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.907598 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-data" pod "b4688f01-eef1-4e8f-9370-05fbecbd6f29" (UID: "b4688f01-eef1-4e8f-9370-05fbecbd6f29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:24.908309 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.908286 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:24.908447 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.908432 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:24.908552 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.908540 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:24.908644 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.908632 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:24.908746 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.908733 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:24.916932 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.916904 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-token" pod "b4688f01-eef1-4e8f-9370-05fbecbd6f29" (UID: "b4688f01-eef1-4e8f-9370-05fbecbd6f29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:24.917329 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.917284 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-envoy" pod "b4688f01-eef1-4e8f-9370-05fbecbd6f29" (UID: "b4688f01-eef1-4e8f-9370-05fbecbd6f29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:24.919116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.919084 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4688f01-eef1-4e8f-9370-05fbecbd6f29-kube-api-access-9l858" pod "b4688f01-eef1-4e8f-9370-05fbecbd6f29" (UID: "b4688f01-eef1-4e8f-9370-05fbecbd6f29"). InnerVolumeSpecName "kube-api-access-9l858". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:24.920198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:24.920164 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-podinfo" pod "b4688f01-eef1-4e8f-9370-05fbecbd6f29" (UID: "b4688f01-eef1-4e8f-9370-05fbecbd6f29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:32:25.009901 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:25.009815 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:25.009901 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:25.009845 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:25.009901 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:25.009854 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4688f01-eef1-4e8f-9370-05fbecbd6f29-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:25.009901 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:25.009863 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l858\" (UniqueName: \"kubernetes.io/projected/b4688f01-eef1-4e8f-9370-05fbecbd6f29-kube-api-access-9l858\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:25.747454 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:25.747423 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" Oct 30 21:32:25.747454 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:25.747441 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc" event={"ID":"b4688f01-eef1-4e8f-9370-05fbecbd6f29","Type":"ContainerDied","Data":"ad8a3be0a8f71d0723509f0e65a2ae05887a5d8b302798d0600a8c935547e792"} Oct 30 21:32:25.747748 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:25.747537 4926 scope.go:122] "RemoveContainer" containerID="6d1357db6dfd71442df6f57b27ee4b780b554ae4a10551396661ffac4691514e" Oct 30 21:32:25.825689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:25.825634 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc"] Oct 30 21:32:25.833750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:25.833714 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-79477e-openshift-default-55bb567445-tf4lc"] Oct 30 21:32:27.021393 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:27.021348 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4688f01-eef1-4e8f-9370-05fbecbd6f29" path="/var/lib/kubelet/pods/b4688f01-eef1-4e8f-9370-05fbecbd6f29/volumes" Oct 30 21:32:28.632398 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:28.632359 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht"] Oct 30 21:32:28.633151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:28.633134 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" containerName="istio-proxy" Oct 30 21:32:28.633151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:28.633152 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" containerName="istio-proxy" Oct 30 21:32:28.633253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:28.633180 4926 cpu_manager.go:414] "containerMap: removing container" podUID="b4688f01-eef1-4e8f-9370-05fbecbd6f29" containerName="istio-proxy" Oct 30 21:32:28.633253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:28.633187 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4688f01-eef1-4e8f-9370-05fbecbd6f29" containerName="istio-proxy" Oct 30 21:32:28.633323 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:28.633303 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c674d3b2-72ca-4b18-9e7c-c6dcdd0ee7b7" containerName="istio-proxy" Oct 30 21:32:28.633323 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:28.633317 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4688f01-eef1-4e8f-9370-05fbecbd6f29" containerName="istio-proxy" Oct 30 21:32:28.638756 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:28.638732 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht" Oct 30 21:32:28.653559 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:28.653533 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht"] Oct 30 21:32:28.659138 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:28.659113 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:32:28.659273 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:28.659171 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:32:28.696732 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:28.696699 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb4ba3a-47b7-46b8-a256-9784a904e748-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6acc995f-2sxht\" (UID: \"afb4ba3a-47b7-46b8-a256-9784a904e748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht" Oct 30 21:32:28.696986 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:28.696873 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb4ba3a-47b7-46b8-a256-9784a904e748-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6acc995f-2sxht\" (UID: \"afb4ba3a-47b7-46b8-a256-9784a904e748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht" Oct 30 21:32:28.697102 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:28.697080 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmdfq\" (UniqueName: \"kubernetes.io/projected/afb4ba3a-47b7-46b8-a256-9784a904e748-kube-api-access-gmdfq\") pod \"maas-api-revoke-sub-6acc995f-2sxht\" (UID: \"afb4ba3a-47b7-46b8-a256-9784a904e748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht" Oct 30 21:32:28.798373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:28.798338 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb4ba3a-47b7-46b8-a256-9784a904e748-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6acc995f-2sxht\" (UID: \"afb4ba3a-47b7-46b8-a256-9784a904e748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht" Oct 30 21:32:28.798596 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:28.798515 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb4ba3a-47b7-46b8-a256-9784a904e748-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6acc995f-2sxht\" (UID: \"afb4ba3a-47b7-46b8-a256-9784a904e748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht" Oct 30 21:32:28.798672 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:28.798647 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmdfq\" (UniqueName: \"kubernetes.io/projected/afb4ba3a-47b7-46b8-a256-9784a904e748-kube-api-access-gmdfq\") pod \"maas-api-revoke-sub-6acc995f-2sxht\" (UID: \"afb4ba3a-47b7-46b8-a256-9784a904e748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht" Oct 30 21:32:28.799509 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:28.799477 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb4ba3a-47b7-46b8-a256-9784a904e748-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6acc995f-2sxht\" (UID: \"afb4ba3a-47b7-46b8-a256-9784a904e748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht" Oct 30 21:32:28.800986 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:28.800959 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb4ba3a-47b7-46b8-a256-9784a904e748-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6acc995f-2sxht\" (UID: \"afb4ba3a-47b7-46b8-a256-9784a904e748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht" Oct 30 21:32:28.818338 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:28.818316 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmdfq\" (UniqueName: \"kubernetes.io/projected/afb4ba3a-47b7-46b8-a256-9784a904e748-kube-api-access-gmdfq\") pod \"maas-api-revoke-sub-6acc995f-2sxht\" (UID: \"afb4ba3a-47b7-46b8-a256-9784a904e748\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht" Oct 30 21:32:28.955373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:28.955336 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht" Oct 30 21:32:29.268879 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:32:29.268830 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb4ba3a_47b7_46b8_a256_9784a904e748.slice/crio-69a4f447171ca9470bfba5028bf8e4a1a44d3f2d654bd15f3d107d7cdc632a17 WatchSource:0}: Error finding container 69a4f447171ca9470bfba5028bf8e4a1a44d3f2d654bd15f3d107d7cdc632a17: Status 404 returned error can't find the container with id 69a4f447171ca9470bfba5028bf8e4a1a44d3f2d654bd15f3d107d7cdc632a17 Oct 30 21:32:29.270703 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:29.270664 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht"] Oct 30 21:32:29.820708 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:29.820656 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht" event={"ID":"afb4ba3a-47b7-46b8-a256-9784a904e748","Type":"ContainerStarted","Data":"2800712563c8ea00d81a4bdb57683fcd9aae6322a8bc677fac24ee22cafbe612"} Oct 30 21:32:29.821147 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:29.820723 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht" event={"ID":"afb4ba3a-47b7-46b8-a256-9784a904e748","Type":"ContainerStarted","Data":"69a4f447171ca9470bfba5028bf8e4a1a44d3f2d654bd15f3d107d7cdc632a17"} Oct 30 21:32:29.863085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:29.863017 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht" podStartSLOduration=1.863001638 podStartE2EDuration="1.863001638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:29.857707694 +0000 UTC m=+1936.978781440" watchObservedRunningTime="2026-10-30 21:32:29.863001638 +0000 UTC m=+1936.984075383" Oct 30 21:32:30.830771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:30.830685 4926 generic.go:356] "Generic (PLEG): container finished" podID="afb4ba3a-47b7-46b8-a256-9784a904e748" containerID="2800712563c8ea00d81a4bdb57683fcd9aae6322a8bc677fac24ee22cafbe612" exitCode=0 Oct 30 21:32:30.831181 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:30.830778 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht" event={"ID":"afb4ba3a-47b7-46b8-a256-9784a904e748","Type":"ContainerDied","Data":"2800712563c8ea00d81a4bdb57683fcd9aae6322a8bc677fac24ee22cafbe612"} Oct 30 21:32:32.099226 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:32.099143 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht" Oct 30 21:32:32.182583 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:32.182544 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afb4ba3a-47b7-46b8-a256-9784a904e748-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb4ba3a-47b7-46b8-a256-9784a904e748-maas-api-openshift-ca\") pod \"afb4ba3a-47b7-46b8-a256-9784a904e748\" (UID: \"afb4ba3a-47b7-46b8-a256-9784a904e748\") " Oct 30 21:32:32.182795 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:32.182601 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afb4ba3a-47b7-46b8-a256-9784a904e748-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb4ba3a-47b7-46b8-a256-9784a904e748-maas-api-xks-ca\") pod \"afb4ba3a-47b7-46b8-a256-9784a904e748\" (UID: \"afb4ba3a-47b7-46b8-a256-9784a904e748\") " Oct 30 21:32:32.182795 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:32.182632 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb4ba3a-47b7-46b8-a256-9784a904e748-kube-api-access-gmdfq\" (UniqueName: \"kubernetes.io/projected/afb4ba3a-47b7-46b8-a256-9784a904e748-kube-api-access-gmdfq\") pod \"afb4ba3a-47b7-46b8-a256-9784a904e748\" (UID: \"afb4ba3a-47b7-46b8-a256-9784a904e748\") " Oct 30 21:32:32.184201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:32.184162 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb4ba3a-47b7-46b8-a256-9784a904e748-maas-api-openshift-ca" pod "afb4ba3a-47b7-46b8-a256-9784a904e748" (UID: "afb4ba3a-47b7-46b8-a256-9784a904e748"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:32.186609 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:32.186581 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb4ba3a-47b7-46b8-a256-9784a904e748-kube-api-access-gmdfq" pod "afb4ba3a-47b7-46b8-a256-9784a904e748" (UID: "afb4ba3a-47b7-46b8-a256-9784a904e748"). InnerVolumeSpecName "kube-api-access-gmdfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:32.186741 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:32.186694 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afb4ba3a-47b7-46b8-a256-9784a904e748-maas-api-xks-ca" pod "afb4ba3a-47b7-46b8-a256-9784a904e748" (UID: "afb4ba3a-47b7-46b8-a256-9784a904e748"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:32.285259 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:32.285224 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afb4ba3a-47b7-46b8-a256-9784a904e748-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:32.285259 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:32.285256 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afb4ba3a-47b7-46b8-a256-9784a904e748-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:32.285259 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:32.285266 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmdfq\" (UniqueName: \"kubernetes.io/projected/afb4ba3a-47b7-46b8-a256-9784a904e748-kube-api-access-gmdfq\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:32.853633 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:32.853597 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht" Oct 30 21:32:32.853633 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:32.853618 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht" event={"ID":"afb4ba3a-47b7-46b8-a256-9784a904e748","Type":"ContainerDied","Data":"69a4f447171ca9470bfba5028bf8e4a1a44d3f2d654bd15f3d107d7cdc632a17"} Oct 30 21:32:32.853959 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:32.853742 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69a4f447171ca9470bfba5028bf8e4a1a44d3f2d654bd15f3d107d7cdc632a17" Oct 30 21:32:34.258049 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:34.257997 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v"] Oct 30 21:32:34.258479 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:34.258285 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" podUID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" containerName="istio-proxy" containerID="cri-o://049e80c22f213e9950d7554314014368899e69eebe4b63e39cbcf837546027c3" gracePeriod=30 Oct 30 21:32:34.477552 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:34.477514 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687"] Oct 30 21:32:34.477751 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:34.477709 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" podUID="f607dfb4-2eab-4446-9501-1ac70b7129f7" containerName="istio-proxy" containerID="cri-o://c1e147327fbb07d55e1e2e0a9e411227c3db0a4e95991e004be57e2dad1b5e85" gracePeriod=30 Oct 30 21:32:35.523366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:35.523327 4926 patch_prober.go:29] interesting pod/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:32:35.523819 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:35.523392 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" podUID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:32:37.487676 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:32:37.487639 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:32:39.629868 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.623796 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:32:39.711911 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.710511 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-envoy\") pod \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " Oct 30 21:32:39.711911 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.710563 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-workload-certs\") pod \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " Oct 30 21:32:39.711911 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.710638 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-podinfo\") pod \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " Oct 30 21:32:39.711911 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.710697 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-credential-socket\") pod \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " Oct 30 21:32:39.711911 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.710747 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f7d1086-9e69-4241-ad92-cc49bf1bca22-kube-api-access-tcqtd\" (UniqueName: \"kubernetes.io/projected/6f7d1086-9e69-4241-ad92-cc49bf1bca22-kube-api-access-tcqtd\") pod \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " Oct 30 21:32:39.711911 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.710779 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-workload-socket\") pod \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " Oct 30 21:32:39.711911 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.710864 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istiod-ca-cert\") pod \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " Oct 30 21:32:39.711911 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.710903 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-token\" (UniqueName: \"kubernetes.io/projected/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-token\") pod \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " Oct 30 21:32:39.711911 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.710989 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-data\") pod \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\" (UID: \"6f7d1086-9e69-4241-ad92-cc49bf1bca22\") " Oct 30 21:32:39.711911 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.711334 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-credential-socket" pod "6f7d1086-9e69-4241-ad92-cc49bf1bca22" (UID: "6f7d1086-9e69-4241-ad92-cc49bf1bca22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:39.711911 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.711873 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:39.712530 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.712246 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-workload-socket" pod "6f7d1086-9e69-4241-ad92-cc49bf1bca22" (UID: "6f7d1086-9e69-4241-ad92-cc49bf1bca22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:39.712530 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.712384 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-data" pod "6f7d1086-9e69-4241-ad92-cc49bf1bca22" (UID: "6f7d1086-9e69-4241-ad92-cc49bf1bca22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:39.714176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.714137 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-workload-certs" pod "6f7d1086-9e69-4241-ad92-cc49bf1bca22" (UID: "6f7d1086-9e69-4241-ad92-cc49bf1bca22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:39.714704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.714666 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istiod-ca-cert" pod "6f7d1086-9e69-4241-ad92-cc49bf1bca22" (UID: "6f7d1086-9e69-4241-ad92-cc49bf1bca22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:39.727048 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.717402 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-token" pod "6f7d1086-9e69-4241-ad92-cc49bf1bca22" (UID: "6f7d1086-9e69-4241-ad92-cc49bf1bca22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:39.727628 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.727575 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7d1086-9e69-4241-ad92-cc49bf1bca22-kube-api-access-tcqtd" pod "6f7d1086-9e69-4241-ad92-cc49bf1bca22" (UID: "6f7d1086-9e69-4241-ad92-cc49bf1bca22"). InnerVolumeSpecName "kube-api-access-tcqtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:39.735752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.735702 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-podinfo" pod "6f7d1086-9e69-4241-ad92-cc49bf1bca22" (UID: "6f7d1086-9e69-4241-ad92-cc49bf1bca22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:32:39.744195 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.744159 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-envoy" pod "6f7d1086-9e69-4241-ad92-cc49bf1bca22" (UID: "6f7d1086-9e69-4241-ad92-cc49bf1bca22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:39.782136 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.782101 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:32:39.813614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.813389 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f607dfb4-2eab-4446-9501-1ac70b7129f7-kube-api-access-57c87\" (UniqueName: \"kubernetes.io/projected/f607dfb4-2eab-4446-9501-1ac70b7129f7-kube-api-access-57c87\") pod \"f607dfb4-2eab-4446-9501-1ac70b7129f7\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " Oct 30 21:32:39.813614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.813542 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-podinfo\") pod \"f607dfb4-2eab-4446-9501-1ac70b7129f7\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " Oct 30 21:32:39.813614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.813600 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-token\" (UniqueName: \"kubernetes.io/projected/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-token\") pod \"f607dfb4-2eab-4446-9501-1ac70b7129f7\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " Oct 30 21:32:39.813965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.813729 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-credential-socket\") pod \"f607dfb4-2eab-4446-9501-1ac70b7129f7\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " Oct 30 21:32:39.813965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.813845 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-workload-certs\") pod \"f607dfb4-2eab-4446-9501-1ac70b7129f7\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " Oct 30 21:32:39.813965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.813884 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-workload-socket\") pod \"f607dfb4-2eab-4446-9501-1ac70b7129f7\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " Oct 30 21:32:39.813965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.813902 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-envoy\") pod \"f607dfb4-2eab-4446-9501-1ac70b7129f7\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " Oct 30 21:32:39.813965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.813942 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-data\") pod \"f607dfb4-2eab-4446-9501-1ac70b7129f7\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " Oct 30 21:32:39.813965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.813966 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f607dfb4-2eab-4446-9501-1ac70b7129f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f607dfb4-2eab-4446-9501-1ac70b7129f7-istiod-ca-cert\") pod \"f607dfb4-2eab-4446-9501-1ac70b7129f7\" (UID: \"f607dfb4-2eab-4446-9501-1ac70b7129f7\") " Oct 30 21:32:39.826568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.814847 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcqtd\" (UniqueName: \"kubernetes.io/projected/6f7d1086-9e69-4241-ad92-cc49bf1bca22-kube-api-access-tcqtd\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:39.826568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.814899 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:39.826568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.814917 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:39.826568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.814928 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:39.826568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.814939 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:39.826568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.814951 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:39.826568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.814963 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f7d1086-9e69-4241-ad92-cc49bf1bca22-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:39.826568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.814974 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f7d1086-9e69-4241-ad92-cc49bf1bca22-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:39.826568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.815413 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f607dfb4-2eab-4446-9501-1ac70b7129f7-istiod-ca-cert" pod "f607dfb4-2eab-4446-9501-1ac70b7129f7" (UID: "f607dfb4-2eab-4446-9501-1ac70b7129f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:39.826568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.817390 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-workload-certs" pod "f607dfb4-2eab-4446-9501-1ac70b7129f7" (UID: "f607dfb4-2eab-4446-9501-1ac70b7129f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:39.826568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.817587 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-credential-socket" pod "f607dfb4-2eab-4446-9501-1ac70b7129f7" (UID: "f607dfb4-2eab-4446-9501-1ac70b7129f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:39.826568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.817753 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-data" pod "f607dfb4-2eab-4446-9501-1ac70b7129f7" (UID: "f607dfb4-2eab-4446-9501-1ac70b7129f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:39.826568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.817912 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-workload-socket" pod "f607dfb4-2eab-4446-9501-1ac70b7129f7" (UID: "f607dfb4-2eab-4446-9501-1ac70b7129f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:39.829477 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.829446 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-podinfo" pod "f607dfb4-2eab-4446-9501-1ac70b7129f7" (UID: "f607dfb4-2eab-4446-9501-1ac70b7129f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:32:39.830342 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.830299 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-envoy" pod "f607dfb4-2eab-4446-9501-1ac70b7129f7" (UID: "f607dfb4-2eab-4446-9501-1ac70b7129f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:32:39.830848 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.830825 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f607dfb4-2eab-4446-9501-1ac70b7129f7-kube-api-access-57c87" pod "f607dfb4-2eab-4446-9501-1ac70b7129f7" (UID: "f607dfb4-2eab-4446-9501-1ac70b7129f7"). InnerVolumeSpecName "kube-api-access-57c87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:39.831483 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.831459 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-token" pod "f607dfb4-2eab-4446-9501-1ac70b7129f7" (UID: "f607dfb4-2eab-4446-9501-1ac70b7129f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:39.916388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.916348 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:39.916388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.916378 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:39.916388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.916389 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:39.916644 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.916402 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:39.916644 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.916414 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:39.916644 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.916422 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f607dfb4-2eab-4446-9501-1ac70b7129f7-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:39.916644 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.916432 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57c87\" (UniqueName: \"kubernetes.io/projected/f607dfb4-2eab-4446-9501-1ac70b7129f7-kube-api-access-57c87\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:39.916644 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.916440 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:39.916644 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.916449 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f607dfb4-2eab-4446-9501-1ac70b7129f7-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:39.926087 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.926056 4926 generic.go:356] "Generic (PLEG): container finished" podID="f607dfb4-2eab-4446-9501-1ac70b7129f7" containerID="c1e147327fbb07d55e1e2e0a9e411227c3db0a4e95991e004be57e2dad1b5e85" exitCode=0 Oct 30 21:32:39.926276 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.926101 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" Oct 30 21:32:39.926276 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.926139 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" event={"ID":"f607dfb4-2eab-4446-9501-1ac70b7129f7","Type":"ContainerDied","Data":"c1e147327fbb07d55e1e2e0a9e411227c3db0a4e95991e004be57e2dad1b5e85"} Oct 30 21:32:39.926276 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.926201 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687" event={"ID":"f607dfb4-2eab-4446-9501-1ac70b7129f7","Type":"ContainerDied","Data":"13213c52ed671612e5a8026c9935f456531c435dce0b5ad36470a460fba52a2d"} Oct 30 21:32:39.926276 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.926230 4926 scope.go:122] "RemoveContainer" containerID="c1e147327fbb07d55e1e2e0a9e411227c3db0a4e95991e004be57e2dad1b5e85" Oct 30 21:32:39.928364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.928338 4926 generic.go:356] "Generic (PLEG): container finished" podID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" containerID="049e80c22f213e9950d7554314014368899e69eebe4b63e39cbcf837546027c3" exitCode=0 Oct 30 21:32:39.928476 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.928383 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" event={"ID":"6f7d1086-9e69-4241-ad92-cc49bf1bca22","Type":"ContainerDied","Data":"049e80c22f213e9950d7554314014368899e69eebe4b63e39cbcf837546027c3"} Oct 30 21:32:39.928476 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.928428 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" event={"ID":"6f7d1086-9e69-4241-ad92-cc49bf1bca22","Type":"ContainerDied","Data":"634ac37cfd9e2606ed28375db90f085a63d7e28079bd97ea2776d5041624d9fa"} Oct 30 21:32:39.928476 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.928386 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v" Oct 30 21:32:39.939866 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.939848 4926 scope.go:122] "RemoveContainer" containerID="c1e147327fbb07d55e1e2e0a9e411227c3db0a4e95991e004be57e2dad1b5e85" Oct 30 21:32:39.940246 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:32:39.940223 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1e147327fbb07d55e1e2e0a9e411227c3db0a4e95991e004be57e2dad1b5e85\": container with ID starting with c1e147327fbb07d55e1e2e0a9e411227c3db0a4e95991e004be57e2dad1b5e85 not found: ID does not exist" containerID="c1e147327fbb07d55e1e2e0a9e411227c3db0a4e95991e004be57e2dad1b5e85" Oct 30 21:32:39.940327 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.940254 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1e147327fbb07d55e1e2e0a9e411227c3db0a4e95991e004be57e2dad1b5e85"} err="failed to get container status \"c1e147327fbb07d55e1e2e0a9e411227c3db0a4e95991e004be57e2dad1b5e85\": rpc error: code = NotFound desc = could not find container \"c1e147327fbb07d55e1e2e0a9e411227c3db0a4e95991e004be57e2dad1b5e85\": container with ID starting with c1e147327fbb07d55e1e2e0a9e411227c3db0a4e95991e004be57e2dad1b5e85 not found: ID does not exist" Oct 30 21:32:39.940327 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.940272 4926 scope.go:122] "RemoveContainer" containerID="049e80c22f213e9950d7554314014368899e69eebe4b63e39cbcf837546027c3" Oct 30 21:32:39.965287 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.965247 4926 scope.go:122] "RemoveContainer" containerID="049e80c22f213e9950d7554314014368899e69eebe4b63e39cbcf837546027c3" Oct 30 21:32:39.965688 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:32:39.965666 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"049e80c22f213e9950d7554314014368899e69eebe4b63e39cbcf837546027c3\": container with ID starting with 049e80c22f213e9950d7554314014368899e69eebe4b63e39cbcf837546027c3 not found: ID does not exist" containerID="049e80c22f213e9950d7554314014368899e69eebe4b63e39cbcf837546027c3" Oct 30 21:32:39.965756 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.965698 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"049e80c22f213e9950d7554314014368899e69eebe4b63e39cbcf837546027c3"} err="failed to get container status \"049e80c22f213e9950d7554314014368899e69eebe4b63e39cbcf837546027c3\": rpc error: code = NotFound desc = could not find container \"049e80c22f213e9950d7554314014368899e69eebe4b63e39cbcf837546027c3\": container with ID starting with 049e80c22f213e9950d7554314014368899e69eebe4b63e39cbcf837546027c3 not found: ID does not exist" Oct 30 21:32:39.987586 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.987555 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687"] Oct 30 21:32:39.994778 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:39.994751 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-2cda44-openshift-default-9c988549c-vt687"] Oct 30 21:32:40.025810 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:40.025773 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v"] Oct 30 21:32:40.031446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:40.031420 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-839163-openshift-default-ff4d9b8f-jmz2v"] Oct 30 21:32:41.018907 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:41.018868 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" path="/var/lib/kubelet/pods/6f7d1086-9e69-4241-ad92-cc49bf1bca22/volumes" Oct 30 21:32:41.019628 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:41.019607 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f607dfb4-2eab-4446-9501-1ac70b7129f7" path="/var/lib/kubelet/pods/f607dfb4-2eab-4446-9501-1ac70b7129f7/volumes" Oct 30 21:32:48.035783 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.035742 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c76951-s96k6"] Oct 30 21:32:48.036542 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.036524 4926 cpu_manager.go:414] "containerMap: removing container" podUID="f607dfb4-2eab-4446-9501-1ac70b7129f7" containerName="istio-proxy" Oct 30 21:32:48.036597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.036544 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f607dfb4-2eab-4446-9501-1ac70b7129f7" containerName="istio-proxy" Oct 30 21:32:48.036597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.036566 4926 cpu_manager.go:414] "containerMap: removing container" podUID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" containerName="istio-proxy" Oct 30 21:32:48.036597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.036571 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" containerName="istio-proxy" Oct 30 21:32:48.036597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.036583 4926 cpu_manager.go:414] "containerMap: removing container" podUID="afb4ba3a-47b7-46b8-a256-9784a904e748" containerName="revoke-keys" Oct 30 21:32:48.036597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.036589 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb4ba3a-47b7-46b8-a256-9784a904e748" containerName="revoke-keys" Oct 30 21:32:48.036774 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.036708 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f7d1086-9e69-4241-ad92-cc49bf1bca22" containerName="istio-proxy" Oct 30 21:32:48.036774 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.036719 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="afb4ba3a-47b7-46b8-a256-9784a904e748" containerName="revoke-keys" Oct 30 21:32:48.036774 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.036733 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="f607dfb4-2eab-4446-9501-1ac70b7129f7" containerName="istio-proxy" Oct 30 21:32:48.042262 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.042241 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c76951-s96k6" Oct 30 21:32:48.073958 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.073919 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:32:48.074869 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.074839 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:32:48.112217 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.106776 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c76951-s96k6"] Oct 30 21:32:48.168134 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.168091 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txvtr\" (UniqueName: \"kubernetes.io/projected/3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c-kube-api-access-txvtr\") pod \"maas-api-revoke-sub-44c76951-s96k6\" (UID: \"3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c76951-s96k6" Oct 30 21:32:48.168355 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.168191 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44c76951-s96k6\" (UID: \"3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c76951-s96k6" Oct 30 21:32:48.168481 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.168454 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44c76951-s96k6\" (UID: \"3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c76951-s96k6" Oct 30 21:32:48.270822 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.270781 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44c76951-s96k6\" (UID: \"3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c76951-s96k6" Oct 30 21:32:48.271054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.270977 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txvtr\" (UniqueName: \"kubernetes.io/projected/3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c-kube-api-access-txvtr\") pod \"maas-api-revoke-sub-44c76951-s96k6\" (UID: \"3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c76951-s96k6" Oct 30 21:32:48.271054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.271003 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44c76951-s96k6\" (UID: \"3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c76951-s96k6" Oct 30 21:32:48.271567 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.271545 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44c76951-s96k6\" (UID: \"3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c76951-s96k6" Oct 30 21:32:48.273568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.273545 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44c76951-s96k6\" (UID: \"3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c76951-s96k6" Oct 30 21:32:48.290770 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.290690 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txvtr\" (UniqueName: \"kubernetes.io/projected/3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c-kube-api-access-txvtr\") pod \"maas-api-revoke-sub-44c76951-s96k6\" (UID: \"3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c76951-s96k6" Oct 30 21:32:48.358299 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.358261 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c76951-s96k6" Oct 30 21:32:48.669648 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:32:48.669606 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8a51a1_e1d3_4ad1_bdcd_018bbf69528c.slice/crio-fde1409d40fcc5a8ee2c6e47b867fb3044d09cf60dec78fab4c45227a8adddb3 WatchSource:0}: Error finding container fde1409d40fcc5a8ee2c6e47b867fb3044d09cf60dec78fab4c45227a8adddb3: Status 404 returned error can't find the container with id fde1409d40fcc5a8ee2c6e47b867fb3044d09cf60dec78fab4c45227a8adddb3 Oct 30 21:32:48.682811 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:48.682772 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c76951-s96k6"] Oct 30 21:32:49.021690 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:49.021648 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c76951-s96k6" event={"ID":"3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c","Type":"ContainerStarted","Data":"fde1409d40fcc5a8ee2c6e47b867fb3044d09cf60dec78fab4c45227a8adddb3"} Oct 30 21:32:50.029134 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:50.029100 4926 generic.go:356] "Generic (PLEG): container finished" podID="3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c" containerID="5b6c7cfe0cf24edd5499a5250f3933498800cd1342032db868d3521f7a603a57" exitCode=0 Oct 30 21:32:50.029622 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:50.029195 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c76951-s96k6" event={"ID":"3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c","Type":"ContainerDied","Data":"5b6c7cfe0cf24edd5499a5250f3933498800cd1342032db868d3521f7a603a57"} Oct 30 21:32:51.232261 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:51.232227 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c76951-s96k6" Oct 30 21:32:51.428820 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:51.428781 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c-kube-api-access-txvtr\" (UniqueName: \"kubernetes.io/projected/3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c-kube-api-access-txvtr\") pod \"3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c\" (UID: \"3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c\") " Oct 30 21:32:51.429025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:51.429009 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c-maas-api-openshift-ca\") pod \"3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c\" (UID: \"3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c\") " Oct 30 21:32:51.429255 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:51.429234 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c-maas-api-xks-ca\") pod \"3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c\" (UID: \"3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c\") " Oct 30 21:32:51.429388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:51.429364 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c-maas-api-openshift-ca" pod "3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c" (UID: "3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:32:51.429837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:51.429818 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:51.431269 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:51.431241 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c-kube-api-access-txvtr" pod "3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c" (UID: "3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c"). InnerVolumeSpecName "kube-api-access-txvtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:32:51.431380 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:51.431296 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c-maas-api-xks-ca" pod "3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c" (UID: "3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:32:51.531117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:51.531078 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:51.531117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:51.531111 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txvtr\" (UniqueName: \"kubernetes.io/projected/3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c-kube-api-access-txvtr\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:32:52.046840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:52.046805 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c76951-s96k6" event={"ID":"3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c","Type":"ContainerDied","Data":"fde1409d40fcc5a8ee2c6e47b867fb3044d09cf60dec78fab4c45227a8adddb3"} Oct 30 21:32:52.047047 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:52.046860 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fde1409d40fcc5a8ee2c6e47b867fb3044d09cf60dec78fab4c45227a8adddb3" Oct 30 21:32:52.047047 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:32:52.046814 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44c76951-s96k6" Oct 30 21:33:07.079242 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:07.079195 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:33:07.495492 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:33:07.495449 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:33:15.760860 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:15.760824 4926 scope.go:122] "RemoveContainer" containerID="57d3e66db12997850149ebc1afadf57788a12f8e068ad9985fb0a603d08fe1a3" Oct 30 21:33:15.773822 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:15.773797 4926 scope.go:122] "RemoveContainer" containerID="f298dda00e2d18651a858ffa0a6f7e9b6545309e9aba8421b24a9745189933c7" Oct 30 21:33:17.893106 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:17.893065 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw"] Oct 30 21:33:17.893648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:17.893399 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" podUID="4bbaeb83-ec01-4556-b964-c56981dc1c5c" containerName="istio-proxy" containerID="cri-o://1313412c3b98d6ec1d26797e636bfa4c90e104b183461de65ddfc278395c2b8d" gracePeriod=30 Oct 30 21:33:23.210149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.210062 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:33:23.350584 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.350546 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bbaeb83-ec01-4556-b964-c56981dc1c5c-kube-api-access-jwzjm\" (UniqueName: \"kubernetes.io/projected/4bbaeb83-ec01-4556-b964-c56981dc1c5c-kube-api-access-jwzjm\") pod \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " Oct 30 21:33:23.350781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.350646 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-workload-certs\") pod \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " Oct 30 21:33:23.350830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.350803 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-token\" (UniqueName: \"kubernetes.io/projected/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-token\") pod \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " Oct 30 21:33:23.350969 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.350947 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-workload-certs" pod "4bbaeb83-ec01-4556-b964-c56981dc1c5c" (UID: "4bbaeb83-ec01-4556-b964-c56981dc1c5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:33:23.351020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.351006 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-data\") pod \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " Oct 30 21:33:23.351084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.351062 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-workload-socket\") pod \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " Oct 30 21:33:23.351128 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.351107 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-podinfo\") pod \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " Oct 30 21:33:23.351187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.351164 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-envoy\") pod \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " Oct 30 21:33:23.351272 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.351254 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-credential-socket\") pod \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " Oct 30 21:33:23.351376 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.351258 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-data" pod "4bbaeb83-ec01-4556-b964-c56981dc1c5c" (UID: "4bbaeb83-ec01-4556-b964-c56981dc1c5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:33:23.351376 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.351367 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istiod-ca-cert\") pod \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\" (UID: \"4bbaeb83-ec01-4556-b964-c56981dc1c5c\") " Oct 30 21:33:23.351502 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.351342 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-workload-socket" pod "4bbaeb83-ec01-4556-b964-c56981dc1c5c" (UID: "4bbaeb83-ec01-4556-b964-c56981dc1c5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:33:23.351586 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.351548 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-credential-socket" pod "4bbaeb83-ec01-4556-b964-c56981dc1c5c" (UID: "4bbaeb83-ec01-4556-b964-c56981dc1c5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:33:23.351745 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.351725 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istiod-ca-cert" pod "4bbaeb83-ec01-4556-b964-c56981dc1c5c" (UID: "4bbaeb83-ec01-4556-b964-c56981dc1c5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:23.352344 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.352322 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:33:23.352408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.352351 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:33:23.352408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.352365 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:33:23.352408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.352382 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:33:23.352408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.352396 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:33:23.353275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.353244 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-token" pod "4bbaeb83-ec01-4556-b964-c56981dc1c5c" (UID: "4bbaeb83-ec01-4556-b964-c56981dc1c5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:23.353456 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.353419 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-envoy" pod "4bbaeb83-ec01-4556-b964-c56981dc1c5c" (UID: "4bbaeb83-ec01-4556-b964-c56981dc1c5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:33:23.353584 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.353459 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bbaeb83-ec01-4556-b964-c56981dc1c5c-kube-api-access-jwzjm" pod "4bbaeb83-ec01-4556-b964-c56981dc1c5c" (UID: "4bbaeb83-ec01-4556-b964-c56981dc1c5c"). InnerVolumeSpecName "kube-api-access-jwzjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:23.353651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.353632 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-podinfo" pod "4bbaeb83-ec01-4556-b964-c56981dc1c5c" (UID: "4bbaeb83-ec01-4556-b964-c56981dc1c5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:33:23.453837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.453801 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwzjm\" (UniqueName: \"kubernetes.io/projected/4bbaeb83-ec01-4556-b964-c56981dc1c5c-kube-api-access-jwzjm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:33:23.453837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.453838 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:33:23.454114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.453850 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:33:23.454114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.453860 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bbaeb83-ec01-4556-b964-c56981dc1c5c-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:33:23.454114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.453943 4926 generic.go:356] "Generic (PLEG): container finished" podID="4bbaeb83-ec01-4556-b964-c56981dc1c5c" containerID="1313412c3b98d6ec1d26797e636bfa4c90e104b183461de65ddfc278395c2b8d" exitCode=0 Oct 30 21:33:23.454114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.453987 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" Oct 30 21:33:23.454114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.454023 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" event={"ID":"4bbaeb83-ec01-4556-b964-c56981dc1c5c","Type":"ContainerDied","Data":"1313412c3b98d6ec1d26797e636bfa4c90e104b183461de65ddfc278395c2b8d"} Oct 30 21:33:23.454285 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.454118 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw" event={"ID":"4bbaeb83-ec01-4556-b964-c56981dc1c5c","Type":"ContainerDied","Data":"0bfaadc3525203145bccbdf2cccb72c621616360515c1c1f2b0b920af3bc2d22"} Oct 30 21:33:23.454285 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.454141 4926 scope.go:122] "RemoveContainer" containerID="1313412c3b98d6ec1d26797e636bfa4c90e104b183461de65ddfc278395c2b8d" Oct 30 21:33:23.468776 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.468754 4926 scope.go:122] "RemoveContainer" containerID="1313412c3b98d6ec1d26797e636bfa4c90e104b183461de65ddfc278395c2b8d" Oct 30 21:33:23.469186 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:33:23.469161 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1313412c3b98d6ec1d26797e636bfa4c90e104b183461de65ddfc278395c2b8d\": container with ID starting with 1313412c3b98d6ec1d26797e636bfa4c90e104b183461de65ddfc278395c2b8d not found: ID does not exist" containerID="1313412c3b98d6ec1d26797e636bfa4c90e104b183461de65ddfc278395c2b8d" Oct 30 21:33:23.469292 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.469191 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1313412c3b98d6ec1d26797e636bfa4c90e104b183461de65ddfc278395c2b8d"} err="failed to get container status \"1313412c3b98d6ec1d26797e636bfa4c90e104b183461de65ddfc278395c2b8d\": rpc error: code = NotFound desc = could not find container \"1313412c3b98d6ec1d26797e636bfa4c90e104b183461de65ddfc278395c2b8d\": container with ID starting with 1313412c3b98d6ec1d26797e636bfa4c90e104b183461de65ddfc278395c2b8d not found: ID does not exist" Oct 30 21:33:23.505118 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.505078 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw"] Oct 30 21:33:23.512700 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:23.512653 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-58fea9-openshift-default-7b68988877-n67hw"] Oct 30 21:33:25.022018 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:25.020991 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bbaeb83-ec01-4556-b964-c56981dc1c5c" path="/var/lib/kubelet/pods/4bbaeb83-ec01-4556-b964-c56981dc1c5c/volumes" Oct 30 21:33:25.610078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:25.610027 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31999-vtppn"] Oct 30 21:33:25.611151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:25.611127 4926 cpu_manager.go:414] "containerMap: removing container" podUID="4bbaeb83-ec01-4556-b964-c56981dc1c5c" containerName="istio-proxy" Oct 30 21:33:25.611252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:25.611152 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bbaeb83-ec01-4556-b964-c56981dc1c5c" containerName="istio-proxy" Oct 30 21:33:25.611252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:25.611175 4926 cpu_manager.go:414] "containerMap: removing container" podUID="3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c" containerName="revoke-keys" Oct 30 21:33:25.611252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:25.611184 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c" containerName="revoke-keys" Oct 30 21:33:25.611394 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:25.611382 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bbaeb83-ec01-4556-b964-c56981dc1c5c" containerName="istio-proxy" Oct 30 21:33:25.611431 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:25.611399 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c" containerName="revoke-keys" Oct 30 21:33:25.617353 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:25.617328 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31999-vtppn" Oct 30 21:33:25.630293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:25.630256 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31999-vtppn"] Oct 30 21:33:25.634163 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:25.634136 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:33:25.634358 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:25.634338 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:33:25.714148 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:25.714101 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c31999-vtppn\" (UID: \"f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31999-vtppn" Oct 30 21:33:25.714369 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:25.714225 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c31999-vtppn\" (UID: \"f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31999-vtppn" Oct 30 21:33:25.714423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:25.714406 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxg9j\" (UniqueName: \"kubernetes.io/projected/f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f-kube-api-access-gxg9j\") pod \"maas-api-revoke-keys-e2e-inf-a-c31999-vtppn\" (UID: \"f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31999-vtppn" Oct 30 21:33:25.815691 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:25.815649 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c31999-vtppn\" (UID: \"f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31999-vtppn" Oct 30 21:33:25.815911 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:25.815755 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c31999-vtppn\" (UID: \"f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31999-vtppn" Oct 30 21:33:25.815980 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:25.815927 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxg9j\" (UniqueName: \"kubernetes.io/projected/f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f-kube-api-access-gxg9j\") pod \"maas-api-revoke-keys-e2e-inf-a-c31999-vtppn\" (UID: \"f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31999-vtppn" Oct 30 21:33:25.816577 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:25.816550 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c31999-vtppn\" (UID: \"f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31999-vtppn" Oct 30 21:33:25.819172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:25.819148 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c31999-vtppn\" (UID: \"f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31999-vtppn" Oct 30 21:33:25.837334 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:25.837300 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxg9j\" (UniqueName: \"kubernetes.io/projected/f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f-kube-api-access-gxg9j\") pod \"maas-api-revoke-keys-e2e-inf-a-c31999-vtppn\" (UID: \"f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31999-vtppn" Oct 30 21:33:25.951313 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:25.951273 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31999-vtppn" Oct 30 21:33:26.273549 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:26.273519 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31999-vtppn"] Oct 30 21:33:26.275359 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:33:26.275327 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29a2ec8_18c3_4bb5_bf96_f716fb01aa2f.slice/crio-0ae6a14c8607bca25e3b96b1a1d816203884fecd212992a07347df03d4d702b7 WatchSource:0}: Error finding container 0ae6a14c8607bca25e3b96b1a1d816203884fecd212992a07347df03d4d702b7: Status 404 returned error can't find the container with id 0ae6a14c8607bca25e3b96b1a1d816203884fecd212992a07347df03d4d702b7 Oct 30 21:33:26.505156 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:26.505112 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31999-vtppn" event={"ID":"f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f","Type":"ContainerStarted","Data":"0ae6a14c8607bca25e3b96b1a1d816203884fecd212992a07347df03d4d702b7"} Oct 30 21:33:27.518574 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:27.518531 4926 generic.go:356] "Generic (PLEG): container finished" podID="f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f" containerID="dd13ff620306b31da7c7d441028ac50c999f6648c1ce7080429830870f6ee0a5" exitCode=0 Oct 30 21:33:27.519066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:27.518612 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31999-vtppn" event={"ID":"f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f","Type":"ContainerDied","Data":"dd13ff620306b31da7c7d441028ac50c999f6648c1ce7080429830870f6ee0a5"} Oct 30 21:33:28.729943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:28.729915 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31999-vtppn" Oct 30 21:33:28.763920 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:28.763870 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f-kube-api-access-gxg9j\" (UniqueName: \"kubernetes.io/projected/f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f-kube-api-access-gxg9j\") pod \"f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f\" (UID: \"f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f\") " Oct 30 21:33:28.764293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:28.764258 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f-maas-api-xks-ca\") pod \"f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f\" (UID: \"f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f\") " Oct 30 21:33:28.764387 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:28.764319 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f-maas-api-openshift-ca\") pod \"f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f\" (UID: \"f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f\") " Oct 30 21:33:28.765790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:28.765757 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f-maas-api-openshift-ca" pod "f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f" (UID: "f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:33:28.781151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:28.780475 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f-maas-api-xks-ca" pod "f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f" (UID: "f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:33:28.791445 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:28.791298 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f-kube-api-access-gxg9j" pod "f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f" (UID: "f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f"). InnerVolumeSpecName "kube-api-access-gxg9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:33:28.866935 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:28.866895 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:33:28.866935 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:28.866931 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:33:28.866935 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:28.866943 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxg9j\" (UniqueName: \"kubernetes.io/projected/f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f-kube-api-access-gxg9j\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:33:29.539057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:29.538996 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31999-vtppn" Oct 30 21:33:29.539057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:29.539043 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31999-vtppn" event={"ID":"f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f","Type":"ContainerDied","Data":"0ae6a14c8607bca25e3b96b1a1d816203884fecd212992a07347df03d4d702b7"} Oct 30 21:33:29.539299 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:33:29.539099 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ae6a14c8607bca25e3b96b1a1d816203884fecd212992a07347df03d4d702b7" Oct 30 21:33:37.502064 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:33:37.502005 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:34:07.508497 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:34:07.508456 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:34:08.736722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:08.736668 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31999-vtppn"] Oct 30 21:34:08.749320 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:08.749282 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31999-vtppn"] Oct 30 21:34:09.019472 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:09.019385 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f" path="/var/lib/kubelet/pods/f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f/volumes" Oct 30 21:34:15.937683 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:15.937626 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8"] Oct 30 21:34:15.938993 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:15.938956 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" podUID="ebd67b20-66f0-472a-ac9b-43aa93a91264" containerName="istio-proxy" containerID="cri-o://7b41130fe3fd8a3263c6b20eb51bd813e90434097fe9bf7020e2b70fe942dbb6" gracePeriod=30 Oct 30 21:34:16.565967 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:16.565932 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r"] Oct 30 21:34:16.566661 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:16.566635 4926 cpu_manager.go:414] "containerMap: removing container" podUID="f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f" containerName="revoke-keys" Oct 30 21:34:16.566661 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:16.566655 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f" containerName="revoke-keys" Oct 30 21:34:16.566868 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:16.566815 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="f29a2ec8-18c3-4bb5-bf96-f716fb01aa2f" containerName="revoke-keys" Oct 30 21:34:16.572125 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:16.572093 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r" Oct 30 21:34:16.594003 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:16.593967 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r"] Oct 30 21:34:16.596400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:16.596370 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:34:16.596538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:16.596409 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:34:16.698855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:16.698308 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d435b5fb-855a-4c16-b8ed-59126022aaee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r\" (UID: \"d435b5fb-855a-4c16-b8ed-59126022aaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r" Oct 30 21:34:16.698855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:16.698406 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmfc4\" (UniqueName: \"kubernetes.io/projected/d435b5fb-855a-4c16-b8ed-59126022aaee-kube-api-access-gmfc4\") pod \"maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r\" (UID: \"d435b5fb-855a-4c16-b8ed-59126022aaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r" Oct 30 21:34:16.698855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:16.698637 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d435b5fb-855a-4c16-b8ed-59126022aaee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r\" (UID: \"d435b5fb-855a-4c16-b8ed-59126022aaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r" Oct 30 21:34:16.799866 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:16.799822 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d435b5fb-855a-4c16-b8ed-59126022aaee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r\" (UID: \"d435b5fb-855a-4c16-b8ed-59126022aaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r" Oct 30 21:34:16.800107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:16.799958 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d435b5fb-855a-4c16-b8ed-59126022aaee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r\" (UID: \"d435b5fb-855a-4c16-b8ed-59126022aaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r" Oct 30 21:34:16.800107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:16.800002 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmfc4\" (UniqueName: \"kubernetes.io/projected/d435b5fb-855a-4c16-b8ed-59126022aaee-kube-api-access-gmfc4\") pod \"maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r\" (UID: \"d435b5fb-855a-4c16-b8ed-59126022aaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r" Oct 30 21:34:16.800630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:16.800605 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d435b5fb-855a-4c16-b8ed-59126022aaee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r\" (UID: \"d435b5fb-855a-4c16-b8ed-59126022aaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r" Oct 30 21:34:16.802579 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:16.802556 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d435b5fb-855a-4c16-b8ed-59126022aaee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r\" (UID: \"d435b5fb-855a-4c16-b8ed-59126022aaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r" Oct 30 21:34:16.819264 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:16.819185 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmfc4\" (UniqueName: \"kubernetes.io/projected/d435b5fb-855a-4c16-b8ed-59126022aaee-kube-api-access-gmfc4\") pod \"maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r\" (UID: \"d435b5fb-855a-4c16-b8ed-59126022aaee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r" Oct 30 21:34:16.888706 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:16.888660 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r" Oct 30 21:34:17.201612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:17.201578 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r"] Oct 30 21:34:17.203069 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:34:17.203011 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd435b5fb_855a_4c16_b8ed_59126022aaee.slice/crio-bcda2da0401f485f8f1616f5164d6e43a7f47c42e15a2016f67e0b110a65e513 WatchSource:0}: Error finding container bcda2da0401f485f8f1616f5164d6e43a7f47c42e15a2016f67e0b110a65e513: Status 404 returned error can't find the container with id bcda2da0401f485f8f1616f5164d6e43a7f47c42e15a2016f67e0b110a65e513 Oct 30 21:34:18.146641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:18.146599 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r" event={"ID":"d435b5fb-855a-4c16-b8ed-59126022aaee","Type":"ContainerStarted","Data":"340afed749be1b530cf249773db339eb8f85c2a9728aafefb3a670a699ff8b3c"} Oct 30 21:34:18.146843 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:18.146668 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r" event={"ID":"d435b5fb-855a-4c16-b8ed-59126022aaee","Type":"ContainerStarted","Data":"bcda2da0401f485f8f1616f5164d6e43a7f47c42e15a2016f67e0b110a65e513"} Oct 30 21:34:18.186630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:18.186565 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r" podStartSLOduration=2.18654653 podStartE2EDuration="2.18654653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:18.183519762 +0000 UTC m=+2045.304593509" watchObservedRunningTime="2026-10-30 21:34:18.18654653 +0000 UTC m=+2045.307620277" Oct 30 21:34:19.156704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:19.156657 4926 generic.go:356] "Generic (PLEG): container finished" podID="d435b5fb-855a-4c16-b8ed-59126022aaee" containerID="340afed749be1b530cf249773db339eb8f85c2a9728aafefb3a670a699ff8b3c" exitCode=0 Oct 30 21:34:19.157162 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:19.156734 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r" event={"ID":"d435b5fb-855a-4c16-b8ed-59126022aaee","Type":"ContainerDied","Data":"340afed749be1b530cf249773db339eb8f85c2a9728aafefb3a670a699ff8b3c"} Oct 30 21:34:20.356748 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:20.356715 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r" Oct 30 21:34:20.463171 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:20.463080 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d435b5fb-855a-4c16-b8ed-59126022aaee-kube-api-access-gmfc4\" (UniqueName: \"kubernetes.io/projected/d435b5fb-855a-4c16-b8ed-59126022aaee-kube-api-access-gmfc4\") pod \"d435b5fb-855a-4c16-b8ed-59126022aaee\" (UID: \"d435b5fb-855a-4c16-b8ed-59126022aaee\") " Oct 30 21:34:20.463355 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:20.463223 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d435b5fb-855a-4c16-b8ed-59126022aaee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d435b5fb-855a-4c16-b8ed-59126022aaee-maas-api-openshift-ca\") pod \"d435b5fb-855a-4c16-b8ed-59126022aaee\" (UID: \"d435b5fb-855a-4c16-b8ed-59126022aaee\") " Oct 30 21:34:20.463546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:20.463527 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d435b5fb-855a-4c16-b8ed-59126022aaee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d435b5fb-855a-4c16-b8ed-59126022aaee-maas-api-xks-ca\") pod \"d435b5fb-855a-4c16-b8ed-59126022aaee\" (UID: \"d435b5fb-855a-4c16-b8ed-59126022aaee\") " Oct 30 21:34:20.463619 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:20.463558 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d435b5fb-855a-4c16-b8ed-59126022aaee-maas-api-openshift-ca" pod "d435b5fb-855a-4c16-b8ed-59126022aaee" (UID: "d435b5fb-855a-4c16-b8ed-59126022aaee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:34:20.464362 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:20.464337 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d435b5fb-855a-4c16-b8ed-59126022aaee-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:20.465298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:20.465265 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d435b5fb-855a-4c16-b8ed-59126022aaee-kube-api-access-gmfc4" pod "d435b5fb-855a-4c16-b8ed-59126022aaee" (UID: "d435b5fb-855a-4c16-b8ed-59126022aaee"). InnerVolumeSpecName "kube-api-access-gmfc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:20.465656 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:20.465626 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d435b5fb-855a-4c16-b8ed-59126022aaee-maas-api-xks-ca" pod "d435b5fb-855a-4c16-b8ed-59126022aaee" (UID: "d435b5fb-855a-4c16-b8ed-59126022aaee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:34:20.566228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:20.566191 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmfc4\" (UniqueName: \"kubernetes.io/projected/d435b5fb-855a-4c16-b8ed-59126022aaee-kube-api-access-gmfc4\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:20.566228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:20.566222 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d435b5fb-855a-4c16-b8ed-59126022aaee-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:20.600603 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:20.600560 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:34:21.185006 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.184971 4926 generic.go:356] "Generic (PLEG): container finished" podID="ebd67b20-66f0-472a-ac9b-43aa93a91264" containerID="7b41130fe3fd8a3263c6b20eb51bd813e90434097fe9bf7020e2b70fe942dbb6" exitCode=0 Oct 30 21:34:21.185282 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.185073 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" event={"ID":"ebd67b20-66f0-472a-ac9b-43aa93a91264","Type":"ContainerDied","Data":"7b41130fe3fd8a3263c6b20eb51bd813e90434097fe9bf7020e2b70fe942dbb6"} Oct 30 21:34:21.187175 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.187151 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r" Oct 30 21:34:21.187333 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.187143 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r" event={"ID":"d435b5fb-855a-4c16-b8ed-59126022aaee","Type":"ContainerDied","Data":"bcda2da0401f485f8f1616f5164d6e43a7f47c42e15a2016f67e0b110a65e513"} Oct 30 21:34:21.187333 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.187272 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcda2da0401f485f8f1616f5164d6e43a7f47c42e15a2016f67e0b110a65e513" Oct 30 21:34:21.221596 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.221563 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:34:21.389924 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.389873 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebd67b20-66f0-472a-ac9b-43aa93a91264-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd67b20-66f0-472a-ac9b-43aa93a91264-istiod-ca-cert\") pod \"ebd67b20-66f0-472a-ac9b-43aa93a91264\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " Oct 30 21:34:21.391239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.391214 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd67b20-66f0-472a-ac9b-43aa93a91264-kube-api-access-lmdsj\" (UniqueName: \"kubernetes.io/projected/ebd67b20-66f0-472a-ac9b-43aa93a91264-kube-api-access-lmdsj\") pod \"ebd67b20-66f0-472a-ac9b-43aa93a91264\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " Oct 30 21:34:21.392606 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.392555 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-workload-certs\") pod \"ebd67b20-66f0-472a-ac9b-43aa93a91264\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " Oct 30 21:34:21.392838 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.392809 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-workload-socket\") pod \"ebd67b20-66f0-472a-ac9b-43aa93a91264\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " Oct 30 21:34:21.393022 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.393009 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-podinfo\") pod \"ebd67b20-66f0-472a-ac9b-43aa93a91264\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " Oct 30 21:34:21.393174 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.393156 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-data\") pod \"ebd67b20-66f0-472a-ac9b-43aa93a91264\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " Oct 30 21:34:21.393393 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.393378 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-envoy\") pod \"ebd67b20-66f0-472a-ac9b-43aa93a91264\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " Oct 30 21:34:21.393657 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.393641 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-token\" (UniqueName: \"kubernetes.io/projected/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-token\") pod \"ebd67b20-66f0-472a-ac9b-43aa93a91264\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " Oct 30 21:34:21.393870 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.393850 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-credential-socket\") pod \"ebd67b20-66f0-472a-ac9b-43aa93a91264\" (UID: \"ebd67b20-66f0-472a-ac9b-43aa93a91264\") " Oct 30 21:34:21.398106 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.398075 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-credential-socket" pod "ebd67b20-66f0-472a-ac9b-43aa93a91264" (UID: "ebd67b20-66f0-472a-ac9b-43aa93a91264"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:34:21.398712 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.398649 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-workload-socket" pod "ebd67b20-66f0-472a-ac9b-43aa93a91264" (UID: "ebd67b20-66f0-472a-ac9b-43aa93a91264"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:34:21.404076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.403395 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-data" pod "ebd67b20-66f0-472a-ac9b-43aa93a91264" (UID: "ebd67b20-66f0-472a-ac9b-43aa93a91264"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:34:21.404414 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.404329 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd67b20-66f0-472a-ac9b-43aa93a91264-istiod-ca-cert" pod "ebd67b20-66f0-472a-ac9b-43aa93a91264" (UID: "ebd67b20-66f0-472a-ac9b-43aa93a91264"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:34:21.406631 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.406593 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-podinfo" pod "ebd67b20-66f0-472a-ac9b-43aa93a91264" (UID: "ebd67b20-66f0-472a-ac9b-43aa93a91264"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:34:21.407996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.407850 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-envoy" pod "ebd67b20-66f0-472a-ac9b-43aa93a91264" (UID: "ebd67b20-66f0-472a-ac9b-43aa93a91264"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:34:21.407996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.407972 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-token" pod "ebd67b20-66f0-472a-ac9b-43aa93a91264" (UID: "ebd67b20-66f0-472a-ac9b-43aa93a91264"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:21.410175 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.410147 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-workload-certs" pod "ebd67b20-66f0-472a-ac9b-43aa93a91264" (UID: "ebd67b20-66f0-472a-ac9b-43aa93a91264"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:34:21.413466 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.413438 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd67b20-66f0-472a-ac9b-43aa93a91264-kube-api-access-lmdsj" pod "ebd67b20-66f0-472a-ac9b-43aa93a91264" (UID: "ebd67b20-66f0-472a-ac9b-43aa93a91264"). InnerVolumeSpecName "kube-api-access-lmdsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:21.496896 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.496788 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmdsj\" (UniqueName: \"kubernetes.io/projected/ebd67b20-66f0-472a-ac9b-43aa93a91264-kube-api-access-lmdsj\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:21.496896 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.496826 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:21.496896 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.496840 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:21.496896 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.496851 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:21.496896 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.496862 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:21.496896 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.496873 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:21.496896 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.496884 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd67b20-66f0-472a-ac9b-43aa93a91264-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:21.496896 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.496896 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd67b20-66f0-472a-ac9b-43aa93a91264-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:21.497296 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:21.496909 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd67b20-66f0-472a-ac9b-43aa93a91264-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:22.199706 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:22.199669 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" Oct 30 21:34:22.199706 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:22.199670 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8" event={"ID":"ebd67b20-66f0-472a-ac9b-43aa93a91264","Type":"ContainerDied","Data":"c94e2fff8a71f4fbd4a6b2b38ca25b2382392206d61dad0658f19bbe7b0de10d"} Oct 30 21:34:22.199956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:22.199796 4926 scope.go:122] "RemoveContainer" containerID="7b41130fe3fd8a3263c6b20eb51bd813e90434097fe9bf7020e2b70fe942dbb6" Oct 30 21:34:22.268568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:22.267503 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8"] Oct 30 21:34:22.276829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:22.276785 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c31999-openshift-default-967c87455-pshs8"] Oct 30 21:34:23.018848 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:23.018813 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebd67b20-66f0-472a-ac9b-43aa93a91264" path="/var/lib/kubelet/pods/ebd67b20-66f0-472a-ac9b-43aa93a91264/volumes" Oct 30 21:34:37.491704 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:34:37.491659 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:34:39.160629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:39.160590 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj"] Oct 30 21:34:39.161211 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:39.160878 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" podUID="76bddd4e-d53f-41f6-95a5-bc445d3eedcf" containerName="istio-proxy" containerID="cri-o://9611169f6e7e5447bd81d6281071d4bd8894bec654e1f060ec40271cfdc9e4ae" gracePeriod=30 Oct 30 21:34:44.422181 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.422145 4926 generic.go:356] "Generic (PLEG): container finished" podID="76bddd4e-d53f-41f6-95a5-bc445d3eedcf" containerID="9611169f6e7e5447bd81d6281071d4bd8894bec654e1f060ec40271cfdc9e4ae" exitCode=0 Oct 30 21:34:44.422674 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.422218 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" event={"ID":"76bddd4e-d53f-41f6-95a5-bc445d3eedcf","Type":"ContainerDied","Data":"9611169f6e7e5447bd81d6281071d4bd8894bec654e1f060ec40271cfdc9e4ae"} Oct 30 21:34:44.422674 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.422280 4926 scope.go:122] "RemoveContainer" containerID="a4e7592dd1ff4fbb169797f585775c45b5a7b6e2b55bd3a64890ee48f2d691a0" Oct 30 21:34:44.473659 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.473631 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:34:44.618864 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.618826 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istiod-ca-cert\") pod \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " Oct 30 21:34:44.619109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.618881 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-workload-socket\") pod \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " Oct 30 21:34:44.619109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.618927 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-envoy\") pod \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " Oct 30 21:34:44.619304 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.619276 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-workload-socket" pod "76bddd4e-d53f-41f6-95a5-bc445d3eedcf" (UID: "76bddd4e-d53f-41f6-95a5-bc445d3eedcf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:34:44.619380 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.619321 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istiod-ca-cert" pod "76bddd4e-d53f-41f6-95a5-bc445d3eedcf" (UID: "76bddd4e-d53f-41f6-95a5-bc445d3eedcf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:34:44.619442 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.619376 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-kube-api-access-pgps2\" (UniqueName: \"kubernetes.io/projected/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-kube-api-access-pgps2\") pod \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " Oct 30 21:34:44.619498 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.619442 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-podinfo\") pod \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " Oct 30 21:34:44.619552 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.619509 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-token\" (UniqueName: \"kubernetes.io/projected/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-token\") pod \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " Oct 30 21:34:44.619621 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.619606 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-workload-certs\") pod \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " Oct 30 21:34:44.619770 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.619757 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-data\") pod \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " Oct 30 21:34:44.619851 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.619838 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-credential-socket\") pod \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\" (UID: \"76bddd4e-d53f-41f6-95a5-bc445d3eedcf\") " Oct 30 21:34:44.620291 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.620202 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-workload-certs" pod "76bddd4e-d53f-41f6-95a5-bc445d3eedcf" (UID: "76bddd4e-d53f-41f6-95a5-bc445d3eedcf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:34:44.620456 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.620432 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-credential-socket" pod "76bddd4e-d53f-41f6-95a5-bc445d3eedcf" (UID: "76bddd4e-d53f-41f6-95a5-bc445d3eedcf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:34:44.620531 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.620450 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-data" pod "76bddd4e-d53f-41f6-95a5-bc445d3eedcf" (UID: "76bddd4e-d53f-41f6-95a5-bc445d3eedcf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:34:44.620703 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.620675 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:44.620703 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.620703 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:44.620845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.620712 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:44.620845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.620720 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:44.622358 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.622327 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-podinfo" pod "76bddd4e-d53f-41f6-95a5-bc445d3eedcf" (UID: "76bddd4e-d53f-41f6-95a5-bc445d3eedcf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:34:44.622517 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.622487 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-kube-api-access-pgps2" pod "76bddd4e-d53f-41f6-95a5-bc445d3eedcf" (UID: "76bddd4e-d53f-41f6-95a5-bc445d3eedcf"). InnerVolumeSpecName "kube-api-access-pgps2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:44.622606 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.622573 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-token" pod "76bddd4e-d53f-41f6-95a5-bc445d3eedcf" (UID: "76bddd4e-d53f-41f6-95a5-bc445d3eedcf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:44.622840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.622805 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-envoy" pod "76bddd4e-d53f-41f6-95a5-bc445d3eedcf" (UID: "76bddd4e-d53f-41f6-95a5-bc445d3eedcf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:34:44.722739 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.722647 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:44.722739 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.722682 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:44.722739 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.722692 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:44.722739 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.722701 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgps2\" (UniqueName: \"kubernetes.io/projected/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-kube-api-access-pgps2\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:44.722739 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:44.722714 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bddd4e-d53f-41f6-95a5-bc445d3eedcf-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:45.432212 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:45.432179 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" Oct 30 21:34:45.432212 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:45.432206 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj" event={"ID":"76bddd4e-d53f-41f6-95a5-bc445d3eedcf","Type":"ContainerDied","Data":"5e5c70cf204f4903ab5668b4d00f119aa3978ea2a761840f0b4af3c30a8f7f28"} Oct 30 21:34:45.432704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:45.432261 4926 scope.go:122] "RemoveContainer" containerID="9611169f6e7e5447bd81d6281071d4bd8894bec654e1f060ec40271cfdc9e4ae" Oct 30 21:34:45.494711 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:45.494679 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj"] Oct 30 21:34:45.506318 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:45.506283 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-4kdcj"] Oct 30 21:34:46.719027 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.718980 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c055fc-4bwvq"] Oct 30 21:34:46.719829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.719804 4926 cpu_manager.go:414] "containerMap: removing container" podUID="ebd67b20-66f0-472a-ac9b-43aa93a91264" containerName="istio-proxy" Oct 30 21:34:46.719829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.719828 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebd67b20-66f0-472a-ac9b-43aa93a91264" containerName="istio-proxy" Oct 30 21:34:46.719996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.719867 4926 cpu_manager.go:414] "containerMap: removing container" podUID="76bddd4e-d53f-41f6-95a5-bc445d3eedcf" containerName="istio-proxy" Oct 30 21:34:46.719996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.719872 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bddd4e-d53f-41f6-95a5-bc445d3eedcf" containerName="istio-proxy" Oct 30 21:34:46.719996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.719892 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d435b5fb-855a-4c16-b8ed-59126022aaee" containerName="revoke-keys" Oct 30 21:34:46.719996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.719898 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d435b5fb-855a-4c16-b8ed-59126022aaee" containerName="revoke-keys" Oct 30 21:34:46.719996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.719906 4926 cpu_manager.go:414] "containerMap: removing container" podUID="76bddd4e-d53f-41f6-95a5-bc445d3eedcf" containerName="istio-proxy" Oct 30 21:34:46.719996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.719911 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bddd4e-d53f-41f6-95a5-bc445d3eedcf" containerName="istio-proxy" Oct 30 21:34:46.720321 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.720004 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebd67b20-66f0-472a-ac9b-43aa93a91264" containerName="istio-proxy" Oct 30 21:34:46.720321 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.720017 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="76bddd4e-d53f-41f6-95a5-bc445d3eedcf" containerName="istio-proxy" Oct 30 21:34:46.720321 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.720026 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d435b5fb-855a-4c16-b8ed-59126022aaee" containerName="revoke-keys" Oct 30 21:34:46.725226 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.725202 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c055fc-4bwvq" Oct 30 21:34:46.750190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.750153 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6f43b83-7049-4398-968a-a2267804b3ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2c055fc-4bwvq\" (UID: \"d6f43b83-7049-4398-968a-a2267804b3ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c055fc-4bwvq" Oct 30 21:34:46.750359 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.750197 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ms5s\" (UniqueName: \"kubernetes.io/projected/d6f43b83-7049-4398-968a-a2267804b3ed-kube-api-access-6ms5s\") pod \"maas-api-revoke-sub-d2c055fc-4bwvq\" (UID: \"d6f43b83-7049-4398-968a-a2267804b3ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c055fc-4bwvq" Oct 30 21:34:46.750359 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.750247 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6f43b83-7049-4398-968a-a2267804b3ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2c055fc-4bwvq\" (UID: \"d6f43b83-7049-4398-968a-a2267804b3ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c055fc-4bwvq" Oct 30 21:34:46.751075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.751017 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c055fc-4bwvq"] Oct 30 21:34:46.756188 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.756158 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:34:46.756309 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.756207 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:34:46.852172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.852135 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6f43b83-7049-4398-968a-a2267804b3ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2c055fc-4bwvq\" (UID: \"d6f43b83-7049-4398-968a-a2267804b3ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c055fc-4bwvq" Oct 30 21:34:46.852172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.852170 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ms5s\" (UniqueName: \"kubernetes.io/projected/d6f43b83-7049-4398-968a-a2267804b3ed-kube-api-access-6ms5s\") pod \"maas-api-revoke-sub-d2c055fc-4bwvq\" (UID: \"d6f43b83-7049-4398-968a-a2267804b3ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c055fc-4bwvq" Oct 30 21:34:46.852445 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.852214 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6f43b83-7049-4398-968a-a2267804b3ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2c055fc-4bwvq\" (UID: \"d6f43b83-7049-4398-968a-a2267804b3ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c055fc-4bwvq" Oct 30 21:34:46.852980 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.852959 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6f43b83-7049-4398-968a-a2267804b3ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2c055fc-4bwvq\" (UID: \"d6f43b83-7049-4398-968a-a2267804b3ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c055fc-4bwvq" Oct 30 21:34:46.854584 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.854564 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6f43b83-7049-4398-968a-a2267804b3ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2c055fc-4bwvq\" (UID: \"d6f43b83-7049-4398-968a-a2267804b3ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c055fc-4bwvq" Oct 30 21:34:46.870946 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:46.870911 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ms5s\" (UniqueName: \"kubernetes.io/projected/d6f43b83-7049-4398-968a-a2267804b3ed-kube-api-access-6ms5s\") pod \"maas-api-revoke-sub-d2c055fc-4bwvq\" (UID: \"d6f43b83-7049-4398-968a-a2267804b3ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c055fc-4bwvq" Oct 30 21:34:47.021293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:47.021214 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76bddd4e-d53f-41f6-95a5-bc445d3eedcf" path="/var/lib/kubelet/pods/76bddd4e-d53f-41f6-95a5-bc445d3eedcf/volumes" Oct 30 21:34:47.041633 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:47.041601 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c055fc-4bwvq" Oct 30 21:34:47.361010 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:47.360979 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c055fc-4bwvq"] Oct 30 21:34:47.362189 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:34:47.362164 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f43b83_7049_4398_968a_a2267804b3ed.slice/crio-ec10a397b545e28a20139dd3efd66a59854a79a9063af05f22a1c358d2a3d055 WatchSource:0}: Error finding container ec10a397b545e28a20139dd3efd66a59854a79a9063af05f22a1c358d2a3d055: Status 404 returned error can't find the container with id ec10a397b545e28a20139dd3efd66a59854a79a9063af05f22a1c358d2a3d055 Oct 30 21:34:47.452115 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:47.452080 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c055fc-4bwvq" event={"ID":"d6f43b83-7049-4398-968a-a2267804b3ed","Type":"ContainerStarted","Data":"ec10a397b545e28a20139dd3efd66a59854a79a9063af05f22a1c358d2a3d055"} Oct 30 21:34:48.462612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:48.462571 4926 generic.go:356] "Generic (PLEG): container finished" podID="d6f43b83-7049-4398-968a-a2267804b3ed" containerID="5e7a2bb96ec68de0284f31e51dcbc6a505b00f319c12492d30c4dc1481ebbcbb" exitCode=0 Oct 30 21:34:48.463085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:48.462638 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c055fc-4bwvq" event={"ID":"d6f43b83-7049-4398-968a-a2267804b3ed","Type":"ContainerDied","Data":"5e7a2bb96ec68de0284f31e51dcbc6a505b00f319c12492d30c4dc1481ebbcbb"} Oct 30 21:34:49.655286 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:49.655249 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c055fc-4bwvq" Oct 30 21:34:49.702382 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:49.702346 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f43b83-7049-4398-968a-a2267804b3ed-kube-api-access-6ms5s\" (UniqueName: \"kubernetes.io/projected/d6f43b83-7049-4398-968a-a2267804b3ed-kube-api-access-6ms5s\") pod \"d6f43b83-7049-4398-968a-a2267804b3ed\" (UID: \"d6f43b83-7049-4398-968a-a2267804b3ed\") " Oct 30 21:34:49.702603 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:49.702586 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6f43b83-7049-4398-968a-a2267804b3ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6f43b83-7049-4398-968a-a2267804b3ed-maas-api-openshift-ca\") pod \"d6f43b83-7049-4398-968a-a2267804b3ed\" (UID: \"d6f43b83-7049-4398-968a-a2267804b3ed\") " Oct 30 21:34:49.702992 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:49.702959 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f43b83-7049-4398-968a-a2267804b3ed-maas-api-openshift-ca" pod "d6f43b83-7049-4398-968a-a2267804b3ed" (UID: "d6f43b83-7049-4398-968a-a2267804b3ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:34:49.703313 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:49.703290 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6f43b83-7049-4398-968a-a2267804b3ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6f43b83-7049-4398-968a-a2267804b3ed-maas-api-xks-ca\") pod \"d6f43b83-7049-4398-968a-a2267804b3ed\" (UID: \"d6f43b83-7049-4398-968a-a2267804b3ed\") " Oct 30 21:34:49.704771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:49.704747 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6f43b83-7049-4398-968a-a2267804b3ed-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:49.705157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:49.705134 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f43b83-7049-4398-968a-a2267804b3ed-kube-api-access-6ms5s" pod "d6f43b83-7049-4398-968a-a2267804b3ed" (UID: "d6f43b83-7049-4398-968a-a2267804b3ed"). InnerVolumeSpecName "kube-api-access-6ms5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:34:49.706156 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:49.706129 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6f43b83-7049-4398-968a-a2267804b3ed-maas-api-xks-ca" pod "d6f43b83-7049-4398-968a-a2267804b3ed" (UID: "d6f43b83-7049-4398-968a-a2267804b3ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:34:49.806455 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:49.806349 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6f43b83-7049-4398-968a-a2267804b3ed-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:49.806455 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:49.806385 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ms5s\" (UniqueName: \"kubernetes.io/projected/d6f43b83-7049-4398-968a-a2267804b3ed-kube-api-access-6ms5s\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:34:50.482618 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:50.482584 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c055fc-4bwvq" Oct 30 21:34:50.482809 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:50.482618 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2c055fc-4bwvq" event={"ID":"d6f43b83-7049-4398-968a-a2267804b3ed","Type":"ContainerDied","Data":"ec10a397b545e28a20139dd3efd66a59854a79a9063af05f22a1c358d2a3d055"} Oct 30 21:34:50.482809 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:50.482682 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec10a397b545e28a20139dd3efd66a59854a79a9063af05f22a1c358d2a3d055" Oct 30 21:34:57.620537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:57.620494 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr"] Oct 30 21:34:58.281497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:58.281460 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz"] Oct 30 21:34:58.282259 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:58.282234 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d6f43b83-7049-4398-968a-a2267804b3ed" containerName="revoke-keys" Oct 30 21:34:58.282341 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:58.282260 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f43b83-7049-4398-968a-a2267804b3ed" containerName="revoke-keys" Oct 30 21:34:58.282487 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:58.282475 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="76bddd4e-d53f-41f6-95a5-bc445d3eedcf" containerName="istio-proxy" Oct 30 21:34:58.282524 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:58.282496 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f43b83-7049-4398-968a-a2267804b3ed" containerName="revoke-keys" Oct 30 21:34:58.286605 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:58.286583 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz" Oct 30 21:34:58.300242 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:58.300215 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:34:58.300409 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:58.300273 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:34:58.304184 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:58.304157 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz"] Oct 30 21:34:58.460510 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:58.460474 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11ccb12-e941-4c28-b18a-7aa50595d7a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3667b84c-bnzzz\" (UID: \"b11ccb12-e941-4c28-b18a-7aa50595d7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz" Oct 30 21:34:58.460710 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:58.460641 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11ccb12-e941-4c28-b18a-7aa50595d7a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3667b84c-bnzzz\" (UID: \"b11ccb12-e941-4c28-b18a-7aa50595d7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz" Oct 30 21:34:58.460809 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:58.460793 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlznm\" (UniqueName: \"kubernetes.io/projected/b11ccb12-e941-4c28-b18a-7aa50595d7a7-kube-api-access-vlznm\") pod \"maas-api-revoke-sub-3667b84c-bnzzz\" (UID: \"b11ccb12-e941-4c28-b18a-7aa50595d7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz" Oct 30 21:34:58.574173 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:58.574073 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlznm\" (UniqueName: \"kubernetes.io/projected/b11ccb12-e941-4c28-b18a-7aa50595d7a7-kube-api-access-vlznm\") pod \"maas-api-revoke-sub-3667b84c-bnzzz\" (UID: \"b11ccb12-e941-4c28-b18a-7aa50595d7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz" Oct 30 21:34:58.574580 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:58.574555 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11ccb12-e941-4c28-b18a-7aa50595d7a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3667b84c-bnzzz\" (UID: \"b11ccb12-e941-4c28-b18a-7aa50595d7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz" Oct 30 21:34:58.574834 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:58.574817 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11ccb12-e941-4c28-b18a-7aa50595d7a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3667b84c-bnzzz\" (UID: \"b11ccb12-e941-4c28-b18a-7aa50595d7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz" Oct 30 21:34:58.575897 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:58.575871 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11ccb12-e941-4c28-b18a-7aa50595d7a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3667b84c-bnzzz\" (UID: \"b11ccb12-e941-4c28-b18a-7aa50595d7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz" Oct 30 21:34:58.623208 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:58.623173 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11ccb12-e941-4c28-b18a-7aa50595d7a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3667b84c-bnzzz\" (UID: \"b11ccb12-e941-4c28-b18a-7aa50595d7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz" Oct 30 21:34:58.672054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:58.661275 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlznm\" (UniqueName: \"kubernetes.io/projected/b11ccb12-e941-4c28-b18a-7aa50595d7a7-kube-api-access-vlznm\") pod \"maas-api-revoke-sub-3667b84c-bnzzz\" (UID: \"b11ccb12-e941-4c28-b18a-7aa50595d7a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz" Oct 30 21:34:58.912141 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:58.908281 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz" Oct 30 21:34:59.711564 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:34:59.711487 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz"] Oct 30 21:35:00.728397 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:00.726257 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz" event={"ID":"b11ccb12-e941-4c28-b18a-7aa50595d7a7","Type":"ContainerStarted","Data":"da566c0baf4b966a9b55a07fd551b6e3a0a80580c4361acd3fd4387bac929556"} Oct 30 21:35:00.728397 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:00.726362 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz" event={"ID":"b11ccb12-e941-4c28-b18a-7aa50595d7a7","Type":"ContainerStarted","Data":"1e6d6125a508bf8fd79299df893a509895b2086cc8c7fd1ffba0043f81090276"} Oct 30 21:35:00.839068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:00.837753 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz" podStartSLOduration=2.837735322 podStartE2EDuration="2.837735322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:34:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:00.822819507 +0000 UTC m=+2087.943893254" watchObservedRunningTime="2026-10-30 21:35:00.837735322 +0000 UTC m=+2087.958809071" Oct 30 21:35:01.239174 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:01.239135 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r"] Oct 30 21:35:01.247819 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:01.247786 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-093daa-cvd5r"] Oct 30 21:35:01.738994 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:01.738957 4926 generic.go:356] "Generic (PLEG): container finished" podID="b11ccb12-e941-4c28-b18a-7aa50595d7a7" containerID="da566c0baf4b966a9b55a07fd551b6e3a0a80580c4361acd3fd4387bac929556" exitCode=0 Oct 30 21:35:01.739480 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:01.739074 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz" event={"ID":"b11ccb12-e941-4c28-b18a-7aa50595d7a7","Type":"ContainerDied","Data":"da566c0baf4b966a9b55a07fd551b6e3a0a80580c4361acd3fd4387bac929556"} Oct 30 21:35:02.961512 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:02.961484 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz" Oct 30 21:35:03.053136 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:03.049465 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d435b5fb-855a-4c16-b8ed-59126022aaee" path="/var/lib/kubelet/pods/d435b5fb-855a-4c16-b8ed-59126022aaee/volumes" Oct 30 21:35:03.061799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:03.061763 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b11ccb12-e941-4c28-b18a-7aa50595d7a7-kube-api-access-vlznm\" (UniqueName: \"kubernetes.io/projected/b11ccb12-e941-4c28-b18a-7aa50595d7a7-kube-api-access-vlznm\") pod \"b11ccb12-e941-4c28-b18a-7aa50595d7a7\" (UID: \"b11ccb12-e941-4c28-b18a-7aa50595d7a7\") " Oct 30 21:35:03.062284 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:03.062257 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b11ccb12-e941-4c28-b18a-7aa50595d7a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11ccb12-e941-4c28-b18a-7aa50595d7a7-maas-api-openshift-ca\") pod \"b11ccb12-e941-4c28-b18a-7aa50595d7a7\" (UID: \"b11ccb12-e941-4c28-b18a-7aa50595d7a7\") " Oct 30 21:35:03.062619 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:03.062599 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b11ccb12-e941-4c28-b18a-7aa50595d7a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11ccb12-e941-4c28-b18a-7aa50595d7a7-maas-api-xks-ca\") pod \"b11ccb12-e941-4c28-b18a-7aa50595d7a7\" (UID: \"b11ccb12-e941-4c28-b18a-7aa50595d7a7\") " Oct 30 21:35:03.091054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:03.078490 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b11ccb12-e941-4c28-b18a-7aa50595d7a7-maas-api-xks-ca" pod "b11ccb12-e941-4c28-b18a-7aa50595d7a7" (UID: "b11ccb12-e941-4c28-b18a-7aa50595d7a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:35:03.091054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:03.079650 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b11ccb12-e941-4c28-b18a-7aa50595d7a7-maas-api-openshift-ca" pod "b11ccb12-e941-4c28-b18a-7aa50595d7a7" (UID: "b11ccb12-e941-4c28-b18a-7aa50595d7a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:35:03.091054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:03.079793 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b11ccb12-e941-4c28-b18a-7aa50595d7a7-kube-api-access-vlznm" pod "b11ccb12-e941-4c28-b18a-7aa50595d7a7" (UID: "b11ccb12-e941-4c28-b18a-7aa50595d7a7"). InnerVolumeSpecName "kube-api-access-vlznm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:03.164850 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:03.164803 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlznm\" (UniqueName: \"kubernetes.io/projected/b11ccb12-e941-4c28-b18a-7aa50595d7a7-kube-api-access-vlznm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:03.164850 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:03.164844 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b11ccb12-e941-4c28-b18a-7aa50595d7a7-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:03.165152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:03.164859 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b11ccb12-e941-4c28-b18a-7aa50595d7a7-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:03.758441 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:03.758407 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz" Oct 30 21:35:03.758629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:03.758435 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz" event={"ID":"b11ccb12-e941-4c28-b18a-7aa50595d7a7","Type":"ContainerDied","Data":"1e6d6125a508bf8fd79299df893a509895b2086cc8c7fd1ffba0043f81090276"} Oct 30 21:35:03.758629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:03.758515 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e6d6125a508bf8fd79299df893a509895b2086cc8c7fd1ffba0043f81090276" Oct 30 21:35:07.489254 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:35:07.489209 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:35:07.977471 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:07.977427 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5"] Oct 30 21:35:07.977761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:07.977729 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" podUID="7b2c155a-636e-4760-9585-e8923ea0318a" containerName="istio-proxy" containerID="cri-o://7ecbe8147b41d84af35b1c51a75fcb688faab6af494dbe446bc8bee4595abf01" gracePeriod=30 Oct 30 21:35:10.093635 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:10.093587 4926 patch_prober.go:29] interesting pod/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:35:10.094238 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:10.093651 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" podUID="7b2c155a-636e-4760-9585-e8923ea0318a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:35:12.643654 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:12.643608 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" podUID="6df032f2-153b-48f9-9e31-3ad79d10bcfc" containerName="main" containerID="cri-o://7d79e9f657cb7324eb7086d2e2264738aa0b250dd5c3bf7af7cd8fd29090bd9e" gracePeriod=45 Oct 30 21:35:12.868807 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:12.867220 4926 generic.go:356] "Generic (PLEG): container finished" podID="6df032f2-153b-48f9-9e31-3ad79d10bcfc" containerID="7d79e9f657cb7324eb7086d2e2264738aa0b250dd5c3bf7af7cd8fd29090bd9e" exitCode=0 Oct 30 21:35:12.868807 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:12.867510 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" event={"ID":"6df032f2-153b-48f9-9e31-3ad79d10bcfc","Type":"ContainerDied","Data":"7d79e9f657cb7324eb7086d2e2264738aa0b250dd5c3bf7af7cd8fd29090bd9e"} Oct 30 21:35:12.975010 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:12.974980 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:35:13.133716 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.133668 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df032f2-153b-48f9-9e31-3ad79d10bcfc-kube-api-access-r52cq\" (UniqueName: \"kubernetes.io/projected/6df032f2-153b-48f9-9e31-3ad79d10bcfc-kube-api-access-r52cq\") pod \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " Oct 30 21:35:13.133917 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.133831 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-dshm\") pod \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " Oct 30 21:35:13.133917 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.133870 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-model-cache\") pod \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " Oct 30 21:35:13.133917 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.133912 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-home\" (UniqueName: \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-home\") pod \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " Oct 30 21:35:13.134092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.134021 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6df032f2-153b-48f9-9e31-3ad79d10bcfc-tls-certs\" (UniqueName: \"kubernetes.io/secret/6df032f2-153b-48f9-9e31-3ad79d10bcfc-tls-certs\") pod \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " Oct 30 21:35:13.134144 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.134119 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-tmp-dir\") pod \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\" (UID: \"6df032f2-153b-48f9-9e31-3ad79d10bcfc\") " Oct 30 21:35:13.134254 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.134225 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-home" pod "6df032f2-153b-48f9-9e31-3ad79d10bcfc" (UID: "6df032f2-153b-48f9-9e31-3ad79d10bcfc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:13.134382 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.134360 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-tmp-dir" pod "6df032f2-153b-48f9-9e31-3ad79d10bcfc" (UID: "6df032f2-153b-48f9-9e31-3ad79d10bcfc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:13.134460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.134441 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-model-cache" pod "6df032f2-153b-48f9-9e31-3ad79d10bcfc" (UID: "6df032f2-153b-48f9-9e31-3ad79d10bcfc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:13.135493 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.134946 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:13.135493 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.134969 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:13.135493 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.134982 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:13.136456 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.136428 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-dshm" pod "6df032f2-153b-48f9-9e31-3ad79d10bcfc" (UID: "6df032f2-153b-48f9-9e31-3ad79d10bcfc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:13.136574 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.136552 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df032f2-153b-48f9-9e31-3ad79d10bcfc-kube-api-access-r52cq" pod "6df032f2-153b-48f9-9e31-3ad79d10bcfc" (UID: "6df032f2-153b-48f9-9e31-3ad79d10bcfc"). InnerVolumeSpecName "kube-api-access-r52cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:13.136985 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.136966 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6df032f2-153b-48f9-9e31-3ad79d10bcfc-tls-certs" pod "6df032f2-153b-48f9-9e31-3ad79d10bcfc" (UID: "6df032f2-153b-48f9-9e31-3ad79d10bcfc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:35:13.236496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.236398 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r52cq\" (UniqueName: \"kubernetes.io/projected/6df032f2-153b-48f9-9e31-3ad79d10bcfc-kube-api-access-r52cq\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:13.236496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.236441 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6df032f2-153b-48f9-9e31-3ad79d10bcfc-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:13.236496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.236456 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6df032f2-153b-48f9-9e31-3ad79d10bcfc-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:13.268663 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.268631 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:35:13.337300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.337261 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-workload-socket\") pod \"7b2c155a-636e-4760-9585-e8923ea0318a\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " Oct 30 21:35:13.337509 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.337363 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-istio-data\") pod \"7b2c155a-636e-4760-9585-e8923ea0318a\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " Oct 30 21:35:13.337509 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.337454 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b2c155a-636e-4760-9585-e8923ea0318a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b2c155a-636e-4760-9585-e8923ea0318a-istiod-ca-cert\") pod \"7b2c155a-636e-4760-9585-e8923ea0318a\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " Oct 30 21:35:13.337509 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.337495 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-workload-certs\") pod \"7b2c155a-636e-4760-9585-e8923ea0318a\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " Oct 30 21:35:13.337693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.337540 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-credential-socket\") pod \"7b2c155a-636e-4760-9585-e8923ea0318a\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " Oct 30 21:35:13.337693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.337648 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b2c155a-636e-4760-9585-e8923ea0318a-kube-api-access-x2bp9\" (UniqueName: \"kubernetes.io/projected/7b2c155a-636e-4760-9585-e8923ea0318a-kube-api-access-x2bp9\") pod \"7b2c155a-636e-4760-9585-e8923ea0318a\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " Oct 30 21:35:13.337693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.337676 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b2c155a-636e-4760-9585-e8923ea0318a-istio-token\" (UniqueName: \"kubernetes.io/projected/7b2c155a-636e-4760-9585-e8923ea0318a-istio-token\") pod \"7b2c155a-636e-4760-9585-e8923ea0318a\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " Oct 30 21:35:13.337852 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.337707 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b2c155a-636e-4760-9585-e8923ea0318a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b2c155a-636e-4760-9585-e8923ea0318a-istio-podinfo\") pod \"7b2c155a-636e-4760-9585-e8923ea0318a\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " Oct 30 21:35:13.337852 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.337764 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-istio-envoy\") pod \"7b2c155a-636e-4760-9585-e8923ea0318a\" (UID: \"7b2c155a-636e-4760-9585-e8923ea0318a\") " Oct 30 21:35:13.338701 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.338281 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-workload-certs" pod "7b2c155a-636e-4760-9585-e8923ea0318a" (UID: "7b2c155a-636e-4760-9585-e8923ea0318a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:13.338701 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.338294 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-workload-socket" pod "7b2c155a-636e-4760-9585-e8923ea0318a" (UID: "7b2c155a-636e-4760-9585-e8923ea0318a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:13.338701 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.338536 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-credential-socket" pod "7b2c155a-636e-4760-9585-e8923ea0318a" (UID: "7b2c155a-636e-4760-9585-e8923ea0318a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:13.338701 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.338684 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b2c155a-636e-4760-9585-e8923ea0318a-istiod-ca-cert" pod "7b2c155a-636e-4760-9585-e8923ea0318a" (UID: "7b2c155a-636e-4760-9585-e8923ea0318a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:35:13.339011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.338972 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-istio-data" pod "7b2c155a-636e-4760-9585-e8923ea0318a" (UID: "7b2c155a-636e-4760-9585-e8923ea0318a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:13.339425 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.339340 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:13.339425 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.339367 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:13.339425 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.339381 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b2c155a-636e-4760-9585-e8923ea0318a-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:13.339425 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.339393 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:13.339425 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.339406 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:13.341219 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.341193 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-istio-envoy" pod "7b2c155a-636e-4760-9585-e8923ea0318a" (UID: "7b2c155a-636e-4760-9585-e8923ea0318a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:13.341348 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.341320 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2c155a-636e-4760-9585-e8923ea0318a-kube-api-access-x2bp9" pod "7b2c155a-636e-4760-9585-e8923ea0318a" (UID: "7b2c155a-636e-4760-9585-e8923ea0318a"). InnerVolumeSpecName "kube-api-access-x2bp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:13.342127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.342103 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2c155a-636e-4760-9585-e8923ea0318a-istio-token" pod "7b2c155a-636e-4760-9585-e8923ea0318a" (UID: "7b2c155a-636e-4760-9585-e8923ea0318a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:13.342241 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.342144 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b2c155a-636e-4760-9585-e8923ea0318a-istio-podinfo" pod "7b2c155a-636e-4760-9585-e8923ea0318a" (UID: "7b2c155a-636e-4760-9585-e8923ea0318a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:35:13.441136 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.441103 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2bp9\" (UniqueName: \"kubernetes.io/projected/7b2c155a-636e-4760-9585-e8923ea0318a-kube-api-access-x2bp9\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:13.441136 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.441136 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b2c155a-636e-4760-9585-e8923ea0318a-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:13.441355 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.441148 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b2c155a-636e-4760-9585-e8923ea0318a-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:13.441355 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.441159 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b2c155a-636e-4760-9585-e8923ea0318a-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:13.496186 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.496098 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:35:13.498294 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.498262 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:35:13.881651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.881558 4926 generic.go:356] "Generic (PLEG): container finished" podID="7b2c155a-636e-4760-9585-e8923ea0318a" containerID="7ecbe8147b41d84af35b1c51a75fcb688faab6af494dbe446bc8bee4595abf01" exitCode=0 Oct 30 21:35:13.881651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.881606 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" Oct 30 21:35:13.881651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.881633 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" event={"ID":"7b2c155a-636e-4760-9585-e8923ea0318a","Type":"ContainerDied","Data":"7ecbe8147b41d84af35b1c51a75fcb688faab6af494dbe446bc8bee4595abf01"} Oct 30 21:35:13.882302 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.881708 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5" event={"ID":"7b2c155a-636e-4760-9585-e8923ea0318a","Type":"ContainerDied","Data":"fd3e1de6cd321ee6035947851bc4fa1be82580ccfce6372bd0b715beceea514d"} Oct 30 21:35:13.882302 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.881741 4926 scope.go:122] "RemoveContainer" containerID="7ecbe8147b41d84af35b1c51a75fcb688faab6af494dbe446bc8bee4595abf01" Oct 30 21:35:13.884416 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.884389 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" event={"ID":"6df032f2-153b-48f9-9e31-3ad79d10bcfc","Type":"ContainerDied","Data":"2fd98c033e69b7c292174101eebfa1eaf9a5d5b6b340e9a31a763e4a92c4b247"} Oct 30 21:35:13.884537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.884471 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr" Oct 30 21:35:13.899541 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.899514 4926 scope.go:122] "RemoveContainer" containerID="7ecbe8147b41d84af35b1c51a75fcb688faab6af494dbe446bc8bee4595abf01" Oct 30 21:35:13.900238 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:35:13.900215 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ecbe8147b41d84af35b1c51a75fcb688faab6af494dbe446bc8bee4595abf01\": container with ID starting with 7ecbe8147b41d84af35b1c51a75fcb688faab6af494dbe446bc8bee4595abf01 not found: ID does not exist" containerID="7ecbe8147b41d84af35b1c51a75fcb688faab6af494dbe446bc8bee4595abf01" Oct 30 21:35:13.900366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.900247 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ecbe8147b41d84af35b1c51a75fcb688faab6af494dbe446bc8bee4595abf01"} err="failed to get container status \"7ecbe8147b41d84af35b1c51a75fcb688faab6af494dbe446bc8bee4595abf01\": rpc error: code = NotFound desc = could not find container \"7ecbe8147b41d84af35b1c51a75fcb688faab6af494dbe446bc8bee4595abf01\": container with ID starting with 7ecbe8147b41d84af35b1c51a75fcb688faab6af494dbe446bc8bee4595abf01 not found: ID does not exist" Oct 30 21:35:13.900366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.900271 4926 scope.go:122] "RemoveContainer" containerID="7d79e9f657cb7324eb7086d2e2264738aa0b250dd5c3bf7af7cd8fd29090bd9e" Oct 30 21:35:13.946071 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:35:13.946017 4926 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2c155a_636e_4760_9585_e8923ea0318a.slice/crio-fd3e1de6cd321ee6035947851bc4fa1be82580ccfce6372bd0b715beceea514d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2c155a_636e_4760_9585_e8923ea0318a.slice\": RecentStats: unable to find data in memory cache]" Oct 30 21:35:13.950358 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.950325 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr"] Oct 30 21:35:13.957379 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.957351 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1eab65/test-model-1eab65-kserve-75f9fc9ccf-vxrtr"] Oct 30 21:35:13.989117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.989079 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5"] Oct 30 21:35:13.995215 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:13.995187 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-093daa-openshift-default-7cd449db7c-8njc5"] Oct 30 21:35:15.019584 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:15.019545 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df032f2-153b-48f9-9e31-3ad79d10bcfc" path="/var/lib/kubelet/pods/6df032f2-153b-48f9-9e31-3ad79d10bcfc/volumes" Oct 30 21:35:15.020208 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:15.020191 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b2c155a-636e-4760-9585-e8923ea0318a" path="/var/lib/kubelet/pods/7b2c155a-636e-4760-9585-e8923ea0318a/volumes" Oct 30 21:35:16.649497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.649467 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb"] Oct 30 21:35:16.650149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.650133 4926 cpu_manager.go:414] "containerMap: removing container" podUID="b11ccb12-e941-4c28-b18a-7aa50595d7a7" containerName="revoke-keys" Oct 30 21:35:16.650207 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.650150 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b11ccb12-e941-4c28-b18a-7aa50595d7a7" containerName="revoke-keys" Oct 30 21:35:16.650207 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.650180 4926 cpu_manager.go:414] "containerMap: removing container" podUID="7b2c155a-636e-4760-9585-e8923ea0318a" containerName="istio-proxy" Oct 30 21:35:16.650207 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.650185 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b2c155a-636e-4760-9585-e8923ea0318a" containerName="istio-proxy" Oct 30 21:35:16.650207 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.650198 4926 cpu_manager.go:414] "containerMap: removing container" podUID="6df032f2-153b-48f9-9e31-3ad79d10bcfc" containerName="main" Oct 30 21:35:16.650207 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.650204 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df032f2-153b-48f9-9e31-3ad79d10bcfc" containerName="main" Oct 30 21:35:16.650365 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.650309 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="b11ccb12-e941-4c28-b18a-7aa50595d7a7" containerName="revoke-keys" Oct 30 21:35:16.650365 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.650322 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b2c155a-636e-4760-9585-e8923ea0318a" containerName="istio-proxy" Oct 30 21:35:16.650365 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.650330 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df032f2-153b-48f9-9e31-3ad79d10bcfc" containerName="main" Oct 30 21:35:16.656019 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.655996 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.680336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.676728 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb"] Oct 30 21:35:16.681943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.681915 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-67f8e0-openshift-default-dockercfg-xjmbd\"" Oct 30 21:35:16.816666 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.809945 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-workload-socket\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.816666 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.810141 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-data\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.816666 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.810242 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-envoy\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.816666 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.810309 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-workload-certs\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.816666 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.810390 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eedfd657-1ae2-4911-b92d-a78c44831f18-istiod-ca-cert\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.816666 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.810454 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-podinfo\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.816666 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.810548 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzwts\" (UniqueName: \"kubernetes.io/projected/eedfd657-1ae2-4911-b92d-a78c44831f18-kube-api-access-jzwts\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.816666 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.810579 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-token\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.816666 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.810605 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-credential-socket\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.911682 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.911583 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-data\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.911682 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.911644 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-envoy\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.911938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.911692 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-workload-certs\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.911938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.911756 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eedfd657-1ae2-4911-b92d-a78c44831f18-istiod-ca-cert\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.911938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.911814 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-podinfo\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.911938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.911900 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzwts\" (UniqueName: \"kubernetes.io/projected/eedfd657-1ae2-4911-b92d-a78c44831f18-kube-api-access-jzwts\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.911938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.911931 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-token\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.912246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.911958 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-credential-socket\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.912246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.912019 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-workload-socket\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.912415 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.912391 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-workload-socket\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.912525 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.912502 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-data\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.912622 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.912598 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eedfd657-1ae2-4911-b92d-a78c44831f18-istiod-ca-cert\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.912895 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.912862 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-credential-socket\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.913227 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.913206 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-workload-certs\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.915341 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.915311 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-podinfo\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.916291 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.916269 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-envoy\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.942624 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.942590 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzwts\" (UniqueName: \"kubernetes.io/projected/eedfd657-1ae2-4911-b92d-a78c44831f18-kube-api-access-jzwts\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.948379 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.948350 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-token\") pod \"e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:16.970699 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:16.970667 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:17.307393 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:17.307330 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb"] Oct 30 21:35:17.310515 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:35:17.310483 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeedfd657_1ae2_4911_b92d_a78c44831f18.slice/crio-66bd96caf09ff99f62b2ac0b5932bdc8e8e303b95d392b0f214d781429483cbd WatchSource:0}: Error finding container 66bd96caf09ff99f62b2ac0b5932bdc8e8e303b95d392b0f214d781429483cbd: Status 404 returned error can't find the container with id 66bd96caf09ff99f62b2ac0b5932bdc8e8e303b95d392b0f214d781429483cbd Oct 30 21:35:17.313496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:17.313461 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:35:17.313624 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:17.313531 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:35:17.313624 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:17.313562 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:35:17.927765 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:17.927721 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" event={"ID":"eedfd657-1ae2-4911-b92d-a78c44831f18","Type":"ContainerStarted","Data":"26ed5007a17b3dbe488a558e6701e5178cb0f8c8ed620666efd7895bd146e578"} Oct 30 21:35:17.928202 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:17.927788 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" event={"ID":"eedfd657-1ae2-4911-b92d-a78c44831f18","Type":"ContainerStarted","Data":"66bd96caf09ff99f62b2ac0b5932bdc8e8e303b95d392b0f214d781429483cbd"} Oct 30 21:35:17.971667 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:17.971627 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:18.997588 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:18.996191 4926 patch_prober.go:29] interesting pod/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 21:35:18.997588 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:18.996297 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" podUID="eedfd657-1ae2-4911-b92d-a78c44831f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 21:35:19.976458 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:19.976416 4926 patch_prober.go:29] interesting pod/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 21:35:19.976839 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:19.976809 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" podUID="eedfd657-1ae2-4911-b92d-a78c44831f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 21:35:20.993057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:20.991369 4926 patch_prober.go:29] interesting pod/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 21:35:20.993057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:20.991435 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" podUID="eedfd657-1ae2-4911-b92d-a78c44831f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 21:35:21.988056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:21.986513 4926 patch_prober.go:29] interesting pod/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 21:35:21.988056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:21.986593 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" podUID="eedfd657-1ae2-4911-b92d-a78c44831f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 21:35:22.971435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:22.971396 4926 patch_prober.go:29] interesting pod/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 21:35:22.972079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:22.972023 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" podUID="eedfd657-1ae2-4911-b92d-a78c44831f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 21:35:23.416986 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:23.416930 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" podStartSLOduration=7.41691391 podStartE2EDuration="7.41691391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:18.001598369 +0000 UTC m=+2105.122672135" watchObservedRunningTime="2026-10-30 21:35:23.41691391 +0000 UTC m=+2110.537987655" Oct 30 21:35:23.418079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:23.418018 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0100b84a-8rg8m"] Oct 30 21:35:23.422695 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:23.422674 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100b84a-8rg8m" Oct 30 21:35:23.439760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:23.439732 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0100b84a-8rg8m"] Oct 30 21:35:23.445341 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:23.445313 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:35:23.445548 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:23.445443 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:35:23.635023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:23.634986 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34af379a-15f2-4d6f-a438-f004b9cccb2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0100b84a-8rg8m\" (UID: \"34af379a-15f2-4d6f-a438-f004b9cccb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100b84a-8rg8m" Oct 30 21:35:23.635215 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:23.635025 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k8dc\" (UniqueName: \"kubernetes.io/projected/34af379a-15f2-4d6f-a438-f004b9cccb2e-kube-api-access-8k8dc\") pod \"maas-api-revoke-sub-0100b84a-8rg8m\" (UID: \"34af379a-15f2-4d6f-a438-f004b9cccb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100b84a-8rg8m" Oct 30 21:35:23.635215 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:23.635135 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34af379a-15f2-4d6f-a438-f004b9cccb2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0100b84a-8rg8m\" (UID: \"34af379a-15f2-4d6f-a438-f004b9cccb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100b84a-8rg8m" Oct 30 21:35:23.736709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:23.736609 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34af379a-15f2-4d6f-a438-f004b9cccb2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0100b84a-8rg8m\" (UID: \"34af379a-15f2-4d6f-a438-f004b9cccb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100b84a-8rg8m" Oct 30 21:35:23.736709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:23.736654 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k8dc\" (UniqueName: \"kubernetes.io/projected/34af379a-15f2-4d6f-a438-f004b9cccb2e-kube-api-access-8k8dc\") pod \"maas-api-revoke-sub-0100b84a-8rg8m\" (UID: \"34af379a-15f2-4d6f-a438-f004b9cccb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100b84a-8rg8m" Oct 30 21:35:23.736923 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:23.736725 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34af379a-15f2-4d6f-a438-f004b9cccb2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0100b84a-8rg8m\" (UID: \"34af379a-15f2-4d6f-a438-f004b9cccb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100b84a-8rg8m" Oct 30 21:35:23.737576 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:23.737531 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34af379a-15f2-4d6f-a438-f004b9cccb2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0100b84a-8rg8m\" (UID: \"34af379a-15f2-4d6f-a438-f004b9cccb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100b84a-8rg8m" Oct 30 21:35:23.739746 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:23.739719 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34af379a-15f2-4d6f-a438-f004b9cccb2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0100b84a-8rg8m\" (UID: \"34af379a-15f2-4d6f-a438-f004b9cccb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100b84a-8rg8m" Oct 30 21:35:23.758786 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:23.758761 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k8dc\" (UniqueName: \"kubernetes.io/projected/34af379a-15f2-4d6f-a438-f004b9cccb2e-kube-api-access-8k8dc\") pod \"maas-api-revoke-sub-0100b84a-8rg8m\" (UID: \"34af379a-15f2-4d6f-a438-f004b9cccb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100b84a-8rg8m" Oct 30 21:35:23.983057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:23.975757 4926 patch_prober.go:29] interesting pod/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" start-of-body= Oct 30 21:35:23.983057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:23.975817 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" podUID="eedfd657-1ae2-4911-b92d-a78c44831f18" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.186:15021/healthz/ready\": dial tcp 10.132.0.186:15021: connect: connection refused" Oct 30 21:35:24.038332 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:24.038235 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100b84a-8rg8m" Oct 30 21:35:24.226781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:24.226746 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec2e75d1-rdc6n"] Oct 30 21:35:24.232460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:24.232432 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2e75d1-rdc6n" Oct 30 21:35:24.258348 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:24.258306 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec2e75d1-rdc6n"] Oct 30 21:35:24.346158 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:24.346053 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgth7\" (UniqueName: \"kubernetes.io/projected/56d39bbb-f292-4757-93aa-03fa94705e39-kube-api-access-jgth7\") pod \"maas-api-revoke-sub-ec2e75d1-rdc6n\" (UID: \"56d39bbb-f292-4757-93aa-03fa94705e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2e75d1-rdc6n" Oct 30 21:35:24.346367 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:24.346206 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d39bbb-f292-4757-93aa-03fa94705e39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec2e75d1-rdc6n\" (UID: \"56d39bbb-f292-4757-93aa-03fa94705e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2e75d1-rdc6n" Oct 30 21:35:24.346367 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:24.346294 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d39bbb-f292-4757-93aa-03fa94705e39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec2e75d1-rdc6n\" (UID: \"56d39bbb-f292-4757-93aa-03fa94705e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2e75d1-rdc6n" Oct 30 21:35:24.424895 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:24.424859 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0100b84a-8rg8m"] Oct 30 21:35:24.425882 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:35:24.425851 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34af379a_15f2_4d6f_a438_f004b9cccb2e.slice/crio-a8e03a83c88cf2aa35915572539669cd32b27385ebcb94c8cef54bd1b54981d9 WatchSource:0}: Error finding container a8e03a83c88cf2aa35915572539669cd32b27385ebcb94c8cef54bd1b54981d9: Status 404 returned error can't find the container with id a8e03a83c88cf2aa35915572539669cd32b27385ebcb94c8cef54bd1b54981d9 Oct 30 21:35:24.447851 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:24.447815 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgth7\" (UniqueName: \"kubernetes.io/projected/56d39bbb-f292-4757-93aa-03fa94705e39-kube-api-access-jgth7\") pod \"maas-api-revoke-sub-ec2e75d1-rdc6n\" (UID: \"56d39bbb-f292-4757-93aa-03fa94705e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2e75d1-rdc6n" Oct 30 21:35:24.448024 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:24.447911 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d39bbb-f292-4757-93aa-03fa94705e39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec2e75d1-rdc6n\" (UID: \"56d39bbb-f292-4757-93aa-03fa94705e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2e75d1-rdc6n" Oct 30 21:35:24.448024 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:24.447963 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d39bbb-f292-4757-93aa-03fa94705e39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec2e75d1-rdc6n\" (UID: \"56d39bbb-f292-4757-93aa-03fa94705e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2e75d1-rdc6n" Oct 30 21:35:24.448923 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:24.448893 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d39bbb-f292-4757-93aa-03fa94705e39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec2e75d1-rdc6n\" (UID: \"56d39bbb-f292-4757-93aa-03fa94705e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2e75d1-rdc6n" Oct 30 21:35:24.450671 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:24.450643 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d39bbb-f292-4757-93aa-03fa94705e39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec2e75d1-rdc6n\" (UID: \"56d39bbb-f292-4757-93aa-03fa94705e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2e75d1-rdc6n" Oct 30 21:35:24.474873 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:24.473090 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgth7\" (UniqueName: \"kubernetes.io/projected/56d39bbb-f292-4757-93aa-03fa94705e39-kube-api-access-jgth7\") pod \"maas-api-revoke-sub-ec2e75d1-rdc6n\" (UID: \"56d39bbb-f292-4757-93aa-03fa94705e39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2e75d1-rdc6n" Oct 30 21:35:24.548941 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:24.548904 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2e75d1-rdc6n" Oct 30 21:35:24.887881 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:35:24.887849 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d39bbb_f292_4757_93aa_03fa94705e39.slice/crio-7fd8e0a1ac0e065c276ba1441415e869126d66d9c8c4f3ca615bb3faca283b54 WatchSource:0}: Error finding container 7fd8e0a1ac0e065c276ba1441415e869126d66d9c8c4f3ca615bb3faca283b54: Status 404 returned error can't find the container with id 7fd8e0a1ac0e065c276ba1441415e869126d66d9c8c4f3ca615bb3faca283b54 Oct 30 21:35:24.888367 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:24.888343 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec2e75d1-rdc6n"] Oct 30 21:35:24.977697 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:24.977663 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:24.978172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:24.978150 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:24.978976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:24.978950 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:35:25.035782 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:25.035746 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2e75d1-rdc6n" event={"ID":"56d39bbb-f292-4757-93aa-03fa94705e39","Type":"ContainerStarted","Data":"7fd8e0a1ac0e065c276ba1441415e869126d66d9c8c4f3ca615bb3faca283b54"} Oct 30 21:35:25.037647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:25.037614 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100b84a-8rg8m" event={"ID":"34af379a-15f2-4d6f-a438-f004b9cccb2e","Type":"ContainerStarted","Data":"3fb7e33adf28861a7b7db1f913cd3a82a399e72963220774091d06a4b41a3fe7"} Oct 30 21:35:25.037790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:25.037660 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100b84a-8rg8m" event={"ID":"34af379a-15f2-4d6f-a438-f004b9cccb2e","Type":"ContainerStarted","Data":"a8e03a83c88cf2aa35915572539669cd32b27385ebcb94c8cef54bd1b54981d9"} Oct 30 21:35:26.047045 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:26.047008 4926 generic.go:356] "Generic (PLEG): container finished" podID="34af379a-15f2-4d6f-a438-f004b9cccb2e" containerID="3fb7e33adf28861a7b7db1f913cd3a82a399e72963220774091d06a4b41a3fe7" exitCode=0 Oct 30 21:35:26.047567 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:26.047107 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100b84a-8rg8m" event={"ID":"34af379a-15f2-4d6f-a438-f004b9cccb2e","Type":"ContainerDied","Data":"3fb7e33adf28861a7b7db1f913cd3a82a399e72963220774091d06a4b41a3fe7"} Oct 30 21:35:26.048968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:26.048945 4926 generic.go:356] "Generic (PLEG): container finished" podID="56d39bbb-f292-4757-93aa-03fa94705e39" containerID="bfdea7644e6c41ac2645bad7e6635397ef8b6bc22af4148c03a8075eca18c395" exitCode=0 Oct 30 21:35:26.049081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:26.049024 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2e75d1-rdc6n" event={"ID":"56d39bbb-f292-4757-93aa-03fa94705e39","Type":"ContainerDied","Data":"bfdea7644e6c41ac2645bad7e6635397ef8b6bc22af4148c03a8075eca18c395"} Oct 30 21:35:27.060807 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.060772 4926 generic.go:356] "Generic (PLEG): container finished" podID="1f0cfe45-affa-48ba-aeea-1f2d5681a022" containerID="17d48eb1b400e377c8f193cc7bef1b9c28f652b8f04c190ccf5282160320f2c7" exitCode=137 Oct 30 21:35:27.061228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.060848 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s" event={"ID":"1f0cfe45-affa-48ba-aeea-1f2d5681a022","Type":"ContainerDied","Data":"17d48eb1b400e377c8f193cc7bef1b9c28f652b8f04c190ccf5282160320f2c7"} Oct 30 21:35:27.061418 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.061398 4926 scope.go:122] "RemoveContainer" containerID="17d48eb1b400e377c8f193cc7bef1b9c28f652b8f04c190ccf5282160320f2c7" Oct 30 21:35:27.348641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.348570 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2e75d1-rdc6n" Oct 30 21:35:27.353708 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.353685 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100b84a-8rg8m" Oct 30 21:35:27.398283 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.398248 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56d39bbb-f292-4757-93aa-03fa94705e39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d39bbb-f292-4757-93aa-03fa94705e39-maas-api-openshift-ca\") pod \"56d39bbb-f292-4757-93aa-03fa94705e39\" (UID: \"56d39bbb-f292-4757-93aa-03fa94705e39\") " Oct 30 21:35:27.398460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.398321 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34af379a-15f2-4d6f-a438-f004b9cccb2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34af379a-15f2-4d6f-a438-f004b9cccb2e-maas-api-xks-ca\") pod \"34af379a-15f2-4d6f-a438-f004b9cccb2e\" (UID: \"34af379a-15f2-4d6f-a438-f004b9cccb2e\") " Oct 30 21:35:27.398460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.398435 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56d39bbb-f292-4757-93aa-03fa94705e39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d39bbb-f292-4757-93aa-03fa94705e39-maas-api-xks-ca\") pod \"56d39bbb-f292-4757-93aa-03fa94705e39\" (UID: \"56d39bbb-f292-4757-93aa-03fa94705e39\") " Oct 30 21:35:27.398568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.398527 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d39bbb-f292-4757-93aa-03fa94705e39-kube-api-access-jgth7\" (UniqueName: \"kubernetes.io/projected/56d39bbb-f292-4757-93aa-03fa94705e39-kube-api-access-jgth7\") pod \"56d39bbb-f292-4757-93aa-03fa94705e39\" (UID: \"56d39bbb-f292-4757-93aa-03fa94705e39\") " Oct 30 21:35:27.398633 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.398614 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34af379a-15f2-4d6f-a438-f004b9cccb2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34af379a-15f2-4d6f-a438-f004b9cccb2e-maas-api-openshift-ca\") pod \"34af379a-15f2-4d6f-a438-f004b9cccb2e\" (UID: \"34af379a-15f2-4d6f-a438-f004b9cccb2e\") " Oct 30 21:35:27.398672 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.398622 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d39bbb-f292-4757-93aa-03fa94705e39-maas-api-openshift-ca" pod "56d39bbb-f292-4757-93aa-03fa94705e39" (UID: "56d39bbb-f292-4757-93aa-03fa94705e39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:35:27.398726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.398683 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34af379a-15f2-4d6f-a438-f004b9cccb2e-kube-api-access-8k8dc\" (UniqueName: \"kubernetes.io/projected/34af379a-15f2-4d6f-a438-f004b9cccb2e-kube-api-access-8k8dc\") pod \"34af379a-15f2-4d6f-a438-f004b9cccb2e\" (UID: \"34af379a-15f2-4d6f-a438-f004b9cccb2e\") " Oct 30 21:35:27.399011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.398985 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34af379a-15f2-4d6f-a438-f004b9cccb2e-maas-api-openshift-ca" pod "34af379a-15f2-4d6f-a438-f004b9cccb2e" (UID: "34af379a-15f2-4d6f-a438-f004b9cccb2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:35:27.399428 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.399407 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56d39bbb-f292-4757-93aa-03fa94705e39-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:27.399428 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.399429 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34af379a-15f2-4d6f-a438-f004b9cccb2e-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:27.400737 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.400713 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d39bbb-f292-4757-93aa-03fa94705e39-kube-api-access-jgth7" pod "56d39bbb-f292-4757-93aa-03fa94705e39" (UID: "56d39bbb-f292-4757-93aa-03fa94705e39"). InnerVolumeSpecName "kube-api-access-jgth7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:27.400814 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.400774 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56d39bbb-f292-4757-93aa-03fa94705e39-maas-api-xks-ca" pod "56d39bbb-f292-4757-93aa-03fa94705e39" (UID: "56d39bbb-f292-4757-93aa-03fa94705e39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:35:27.400983 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.400968 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34af379a-15f2-4d6f-a438-f004b9cccb2e-maas-api-xks-ca" pod "34af379a-15f2-4d6f-a438-f004b9cccb2e" (UID: "34af379a-15f2-4d6f-a438-f004b9cccb2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:35:27.401210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.401190 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34af379a-15f2-4d6f-a438-f004b9cccb2e-kube-api-access-8k8dc" pod "34af379a-15f2-4d6f-a438-f004b9cccb2e" (UID: "34af379a-15f2-4d6f-a438-f004b9cccb2e"). InnerVolumeSpecName "kube-api-access-8k8dc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:27.500794 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.500756 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56d39bbb-f292-4757-93aa-03fa94705e39-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:27.500794 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.500789 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgth7\" (UniqueName: \"kubernetes.io/projected/56d39bbb-f292-4757-93aa-03fa94705e39-kube-api-access-jgth7\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:27.500794 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.500801 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k8dc\" (UniqueName: \"kubernetes.io/projected/34af379a-15f2-4d6f-a438-f004b9cccb2e-kube-api-access-8k8dc\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:27.501054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:27.500810 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34af379a-15f2-4d6f-a438-f004b9cccb2e-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:28.074291 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:28.074257 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s" event={"ID":"1f0cfe45-affa-48ba-aeea-1f2d5681a022","Type":"ContainerStarted","Data":"8693ec263bb3e9720e99db955db1f076c891ffcc3d075fe3df1a4ac45fb66f28"} Oct 30 21:35:28.074750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:28.074482 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s" Oct 30 21:35:28.076149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:28.076124 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2e75d1-rdc6n" Oct 30 21:35:28.076281 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:28.076179 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec2e75d1-rdc6n" event={"ID":"56d39bbb-f292-4757-93aa-03fa94705e39","Type":"ContainerDied","Data":"7fd8e0a1ac0e065c276ba1441415e869126d66d9c8c4f3ca615bb3faca283b54"} Oct 30 21:35:28.076281 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:28.076226 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fd8e0a1ac0e065c276ba1441415e869126d66d9c8c4f3ca615bb3faca283b54" Oct 30 21:35:28.077811 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:28.077787 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100b84a-8rg8m" event={"ID":"34af379a-15f2-4d6f-a438-f004b9cccb2e","Type":"ContainerDied","Data":"a8e03a83c88cf2aa35915572539669cd32b27385ebcb94c8cef54bd1b54981d9"} Oct 30 21:35:28.077938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:28.077827 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0100b84a-8rg8m" Oct 30 21:35:28.077938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:28.077830 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8e03a83c88cf2aa35915572539669cd32b27385ebcb94c8cef54bd1b54981d9" Oct 30 21:35:32.055971 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:32.055933 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp"] Oct 30 21:35:32.392110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:32.392004 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh"] Oct 30 21:35:32.392744 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:32.392727 4926 cpu_manager.go:414] "containerMap: removing container" podUID="56d39bbb-f292-4757-93aa-03fa94705e39" containerName="revoke-keys" Oct 30 21:35:32.392744 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:32.392745 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d39bbb-f292-4757-93aa-03fa94705e39" containerName="revoke-keys" Oct 30 21:35:32.392857 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:32.392764 4926 cpu_manager.go:414] "containerMap: removing container" podUID="34af379a-15f2-4d6f-a438-f004b9cccb2e" containerName="revoke-keys" Oct 30 21:35:32.392857 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:32.392769 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34af379a-15f2-4d6f-a438-f004b9cccb2e" containerName="revoke-keys" Oct 30 21:35:32.392932 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:32.392880 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="34af379a-15f2-4d6f-a438-f004b9cccb2e" containerName="revoke-keys" Oct 30 21:35:32.392932 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:32.392899 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d39bbb-f292-4757-93aa-03fa94705e39" containerName="revoke-keys" Oct 30 21:35:32.396958 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:32.396922 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh" Oct 30 21:35:32.416376 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:32.416342 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:35:32.416661 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:32.416639 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:35:32.420698 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:32.420659 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh"] Oct 30 21:35:32.488731 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:32.488672 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d947c8-b5e0-4d7c-8517-c2fd7984424d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh\" (UID: \"24d947c8-b5e0-4d7c-8517-c2fd7984424d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh" Oct 30 21:35:32.488982 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:32.488956 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d947c8-b5e0-4d7c-8517-c2fd7984424d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh\" (UID: \"24d947c8-b5e0-4d7c-8517-c2fd7984424d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh" Oct 30 21:35:32.489085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:32.488999 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsj5t\" (UniqueName: \"kubernetes.io/projected/24d947c8-b5e0-4d7c-8517-c2fd7984424d-kube-api-access-vsj5t\") pod \"maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh\" (UID: \"24d947c8-b5e0-4d7c-8517-c2fd7984424d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh" Oct 30 21:35:32.591028 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:32.590980 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d947c8-b5e0-4d7c-8517-c2fd7984424d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh\" (UID: \"24d947c8-b5e0-4d7c-8517-c2fd7984424d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh" Oct 30 21:35:32.591028 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:32.591069 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsj5t\" (UniqueName: \"kubernetes.io/projected/24d947c8-b5e0-4d7c-8517-c2fd7984424d-kube-api-access-vsj5t\") pod \"maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh\" (UID: \"24d947c8-b5e0-4d7c-8517-c2fd7984424d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh" Oct 30 21:35:32.591328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:32.591289 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d947c8-b5e0-4d7c-8517-c2fd7984424d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh\" (UID: \"24d947c8-b5e0-4d7c-8517-c2fd7984424d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh" Oct 30 21:35:32.591661 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:32.591638 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d947c8-b5e0-4d7c-8517-c2fd7984424d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh\" (UID: \"24d947c8-b5e0-4d7c-8517-c2fd7984424d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh" Oct 30 21:35:32.593708 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:32.593681 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d947c8-b5e0-4d7c-8517-c2fd7984424d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh\" (UID: \"24d947c8-b5e0-4d7c-8517-c2fd7984424d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh" Oct 30 21:35:32.618108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:32.618068 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsj5t\" (UniqueName: \"kubernetes.io/projected/24d947c8-b5e0-4d7c-8517-c2fd7984424d-kube-api-access-vsj5t\") pod \"maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh\" (UID: \"24d947c8-b5e0-4d7c-8517-c2fd7984424d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh" Oct 30 21:35:32.713192 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:32.713160 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh" Oct 30 21:35:33.023518 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:35:33.023485 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d947c8_b5e0_4d7c_8517_c2fd7984424d.slice/crio-7970c049305bbc5f58edfcc5bc8091894c1d89b70598e0f9a71a2f7b1e9c8e26 WatchSource:0}: Error finding container 7970c049305bbc5f58edfcc5bc8091894c1d89b70598e0f9a71a2f7b1e9c8e26: Status 404 returned error can't find the container with id 7970c049305bbc5f58edfcc5bc8091894c1d89b70598e0f9a71a2f7b1e9c8e26 Oct 30 21:35:33.024682 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:33.024652 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh"] Oct 30 21:35:33.125283 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:33.125244 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh" event={"ID":"24d947c8-b5e0-4d7c-8517-c2fd7984424d","Type":"ContainerStarted","Data":"7970c049305bbc5f58edfcc5bc8091894c1d89b70598e0f9a71a2f7b1e9c8e26"} Oct 30 21:35:34.134477 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:34.134440 4926 generic.go:356] "Generic (PLEG): container finished" podID="24d947c8-b5e0-4d7c-8517-c2fd7984424d" containerID="2709840fba0af87eeb0f4f2236e097896ad662ac1cdc2f23c3f90ee5d94d3d2e" exitCode=0 Oct 30 21:35:34.134980 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:34.134523 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh" event={"ID":"24d947c8-b5e0-4d7c-8517-c2fd7984424d","Type":"ContainerDied","Data":"2709840fba0af87eeb0f4f2236e097896ad662ac1cdc2f23c3f90ee5d94d3d2e"} Oct 30 21:35:35.353145 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:35.353110 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh" Oct 30 21:35:35.442140 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:35.442103 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24d947c8-b5e0-4d7c-8517-c2fd7984424d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d947c8-b5e0-4d7c-8517-c2fd7984424d-maas-api-openshift-ca\") pod \"24d947c8-b5e0-4d7c-8517-c2fd7984424d\" (UID: \"24d947c8-b5e0-4d7c-8517-c2fd7984424d\") " Oct 30 21:35:35.442366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:35.442239 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24d947c8-b5e0-4d7c-8517-c2fd7984424d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d947c8-b5e0-4d7c-8517-c2fd7984424d-maas-api-xks-ca\") pod \"24d947c8-b5e0-4d7c-8517-c2fd7984424d\" (UID: \"24d947c8-b5e0-4d7c-8517-c2fd7984424d\") " Oct 30 21:35:35.442366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:35.442276 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24d947c8-b5e0-4d7c-8517-c2fd7984424d-kube-api-access-vsj5t\" (UniqueName: \"kubernetes.io/projected/24d947c8-b5e0-4d7c-8517-c2fd7984424d-kube-api-access-vsj5t\") pod \"24d947c8-b5e0-4d7c-8517-c2fd7984424d\" (UID: \"24d947c8-b5e0-4d7c-8517-c2fd7984424d\") " Oct 30 21:35:35.442640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:35.442607 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d947c8-b5e0-4d7c-8517-c2fd7984424d-maas-api-openshift-ca" pod "24d947c8-b5e0-4d7c-8517-c2fd7984424d" (UID: "24d947c8-b5e0-4d7c-8517-c2fd7984424d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:35:35.443144 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:35.443116 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d947c8-b5e0-4d7c-8517-c2fd7984424d-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:35.445542 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:35.445510 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d947c8-b5e0-4d7c-8517-c2fd7984424d-kube-api-access-vsj5t" pod "24d947c8-b5e0-4d7c-8517-c2fd7984424d" (UID: "24d947c8-b5e0-4d7c-8517-c2fd7984424d"). InnerVolumeSpecName "kube-api-access-vsj5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:35.446199 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:35.445917 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24d947c8-b5e0-4d7c-8517-c2fd7984424d-maas-api-xks-ca" pod "24d947c8-b5e0-4d7c-8517-c2fd7984424d" (UID: "24d947c8-b5e0-4d7c-8517-c2fd7984424d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:35:35.563072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:35.562151 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d947c8-b5e0-4d7c-8517-c2fd7984424d-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:35.563072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:35.562193 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsj5t\" (UniqueName: \"kubernetes.io/projected/24d947c8-b5e0-4d7c-8517-c2fd7984424d-kube-api-access-vsj5t\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:36.152452 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:36.152420 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh" Oct 30 21:35:36.152685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:36.152449 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh" event={"ID":"24d947c8-b5e0-4d7c-8517-c2fd7984424d","Type":"ContainerDied","Data":"7970c049305bbc5f58edfcc5bc8091894c1d89b70598e0f9a71a2f7b1e9c8e26"} Oct 30 21:35:36.152685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:36.152501 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7970c049305bbc5f58edfcc5bc8091894c1d89b70598e0f9a71a2f7b1e9c8e26" Oct 30 21:35:37.488737 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:35:37.488697 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:35:37.652208 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:37.652168 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z"] Oct 30 21:35:37.652415 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:37.652388 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" podUID="9cbcfa4c-ad1b-4110-a069-38c79809bc98" containerName="istio-proxy" containerID="cri-o://fa0f023ed372bf167ac295cb3f6bb66405aa557ac96199396087a4d95ecb9d6c" gracePeriod=30 Oct 30 21:35:39.089887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:39.089853 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s" Oct 30 21:35:41.264750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:41.264704 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-cdm7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:35:41.265316 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:41.264786 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" podUID="9cbcfa4c-ad1b-4110-a069-38c79809bc98" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:35:42.976858 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:42.976822 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:35:43.079497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.079455 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cbcfa4c-ad1b-4110-a069-38c79809bc98-kube-api-access-99d9s\" (UniqueName: \"kubernetes.io/projected/9cbcfa4c-ad1b-4110-a069-38c79809bc98-kube-api-access-99d9s\") pod \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " Oct 30 21:35:43.079698 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.079553 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-data\") pod \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " Oct 30 21:35:43.079698 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.079678 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-envoy\") pod \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " Oct 30 21:35:43.079835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.079706 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-workload-socket\") pod \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " Oct 30 21:35:43.079835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.079745 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-credential-socket\") pod \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " Oct 30 21:35:43.079835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.079774 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-token\" (UniqueName: \"kubernetes.io/projected/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-token\") pod \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " Oct 30 21:35:43.079835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.079822 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-workload-certs\") pod \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " Oct 30 21:35:43.080281 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.079928 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istiod-ca-cert\") pod \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " Oct 30 21:35:43.080281 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.079988 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-podinfo\") pod \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\" (UID: \"9cbcfa4c-ad1b-4110-a069-38c79809bc98\") " Oct 30 21:35:43.082256 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.082226 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cbcfa4c-ad1b-4110-a069-38c79809bc98-kube-api-access-99d9s" pod "9cbcfa4c-ad1b-4110-a069-38c79809bc98" (UID: "9cbcfa4c-ad1b-4110-a069-38c79809bc98"). InnerVolumeSpecName "kube-api-access-99d9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:43.083220 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.083139 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-data" pod "9cbcfa4c-ad1b-4110-a069-38c79809bc98" (UID: "9cbcfa4c-ad1b-4110-a069-38c79809bc98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:43.085102 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.083644 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-workload-socket" pod "9cbcfa4c-ad1b-4110-a069-38c79809bc98" (UID: "9cbcfa4c-ad1b-4110-a069-38c79809bc98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:43.085235 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.083730 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istiod-ca-cert" pod "9cbcfa4c-ad1b-4110-a069-38c79809bc98" (UID: "9cbcfa4c-ad1b-4110-a069-38c79809bc98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:35:43.085342 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.083997 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-workload-certs" pod "9cbcfa4c-ad1b-4110-a069-38c79809bc98" (UID: "9cbcfa4c-ad1b-4110-a069-38c79809bc98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:43.085437 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.084006 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-credential-socket" pod "9cbcfa4c-ad1b-4110-a069-38c79809bc98" (UID: "9cbcfa4c-ad1b-4110-a069-38c79809bc98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:43.089405 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.089380 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-envoy" pod "9cbcfa4c-ad1b-4110-a069-38c79809bc98" (UID: "9cbcfa4c-ad1b-4110-a069-38c79809bc98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:43.089481 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.089435 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-token" pod "9cbcfa4c-ad1b-4110-a069-38c79809bc98" (UID: "9cbcfa4c-ad1b-4110-a069-38c79809bc98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:43.090604 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.090573 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-podinfo" pod "9cbcfa4c-ad1b-4110-a069-38c79809bc98" (UID: "9cbcfa4c-ad1b-4110-a069-38c79809bc98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:35:43.191665 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.191565 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:43.191951 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.191928 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:43.192239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.192186 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:43.192391 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.192361 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:43.192507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.192495 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:43.192637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.192625 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:43.192758 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.192747 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:43.192884 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.192847 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99d9s\" (UniqueName: \"kubernetes.io/projected/9cbcfa4c-ad1b-4110-a069-38c79809bc98-kube-api-access-99d9s\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:43.192996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.192983 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cbcfa4c-ad1b-4110-a069-38c79809bc98-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:43.218239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.218199 4926 generic.go:356] "Generic (PLEG): container finished" podID="9cbcfa4c-ad1b-4110-a069-38c79809bc98" containerID="fa0f023ed372bf167ac295cb3f6bb66405aa557ac96199396087a4d95ecb9d6c" exitCode=0 Oct 30 21:35:43.218430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.218249 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" Oct 30 21:35:43.218430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.218291 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" event={"ID":"9cbcfa4c-ad1b-4110-a069-38c79809bc98","Type":"ContainerDied","Data":"fa0f023ed372bf167ac295cb3f6bb66405aa557ac96199396087a4d95ecb9d6c"} Oct 30 21:35:43.218430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.218370 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z" event={"ID":"9cbcfa4c-ad1b-4110-a069-38c79809bc98","Type":"ContainerDied","Data":"39671291116a48a1b204098de72ac7a92ac7bf74c92a12d8d1f772eef06b3794"} Oct 30 21:35:43.218430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.218407 4926 scope.go:122] "RemoveContainer" containerID="fa0f023ed372bf167ac295cb3f6bb66405aa557ac96199396087a4d95ecb9d6c" Oct 30 21:35:43.233004 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.232979 4926 scope.go:122] "RemoveContainer" containerID="fa0f023ed372bf167ac295cb3f6bb66405aa557ac96199396087a4d95ecb9d6c" Oct 30 21:35:43.233653 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:35:43.233631 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa0f023ed372bf167ac295cb3f6bb66405aa557ac96199396087a4d95ecb9d6c\": container with ID starting with fa0f023ed372bf167ac295cb3f6bb66405aa557ac96199396087a4d95ecb9d6c not found: ID does not exist" containerID="fa0f023ed372bf167ac295cb3f6bb66405aa557ac96199396087a4d95ecb9d6c" Oct 30 21:35:43.233764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.233663 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa0f023ed372bf167ac295cb3f6bb66405aa557ac96199396087a4d95ecb9d6c"} err="failed to get container status \"fa0f023ed372bf167ac295cb3f6bb66405aa557ac96199396087a4d95ecb9d6c\": rpc error: code = NotFound desc = could not find container \"fa0f023ed372bf167ac295cb3f6bb66405aa557ac96199396087a4d95ecb9d6c\": container with ID starting with fa0f023ed372bf167ac295cb3f6bb66405aa557ac96199396087a4d95ecb9d6c not found: ID does not exist" Oct 30 21:35:43.289189 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.289148 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z"] Oct 30 21:35:43.300197 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:43.300159 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-cdm7z"] Oct 30 21:35:45.020951 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:45.020910 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cbcfa4c-ad1b-4110-a069-38c79809bc98" path="/var/lib/kubelet/pods/9cbcfa4c-ad1b-4110-a069-38c79809bc98/volumes" Oct 30 21:35:47.078843 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:47.078782 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" podUID="f05d1479-4fa1-4640-a6f5-59f43084b401" containerName="main" containerID="cri-o://1d1cd4004e14bb6aac1c584def6c23b4ac39f1a6b37c27e7db0bc8fd137b4ccc" gracePeriod=45 Oct 30 21:35:47.304604 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:47.304572 4926 generic.go:356] "Generic (PLEG): container finished" podID="f05d1479-4fa1-4640-a6f5-59f43084b401" containerID="1d1cd4004e14bb6aac1c584def6c23b4ac39f1a6b37c27e7db0bc8fd137b4ccc" exitCode=0 Oct 30 21:35:47.304797 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:47.304653 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" event={"ID":"f05d1479-4fa1-4640-a6f5-59f43084b401","Type":"ContainerDied","Data":"1d1cd4004e14bb6aac1c584def6c23b4ac39f1a6b37c27e7db0bc8fd137b4ccc"} Oct 30 21:35:47.369946 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:47.369916 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:35:47.476792 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:47.476746 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f05d1479-4fa1-4640-a6f5-59f43084b401-kube-api-access-kt7zm\" (UniqueName: \"kubernetes.io/projected/f05d1479-4fa1-4640-a6f5-59f43084b401-kube-api-access-kt7zm\") pod \"f05d1479-4fa1-4640-a6f5-59f43084b401\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " Oct 30 21:35:47.477020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:47.476816 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f05d1479-4fa1-4640-a6f5-59f43084b401-tls-certs\" (UniqueName: \"kubernetes.io/secret/f05d1479-4fa1-4640-a6f5-59f43084b401-tls-certs\") pod \"f05d1479-4fa1-4640-a6f5-59f43084b401\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " Oct 30 21:35:47.477445 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:47.477313 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-model-cache\") pod \"f05d1479-4fa1-4640-a6f5-59f43084b401\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " Oct 30 21:35:47.477870 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:47.477845 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-home\" (UniqueName: \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-home\") pod \"f05d1479-4fa1-4640-a6f5-59f43084b401\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " Oct 30 21:35:47.478006 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:47.477953 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-model-cache" pod "f05d1479-4fa1-4640-a6f5-59f43084b401" (UID: "f05d1479-4fa1-4640-a6f5-59f43084b401"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:47.478114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:47.478099 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-tmp-dir\") pod \"f05d1479-4fa1-4640-a6f5-59f43084b401\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " Oct 30 21:35:47.478524 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:47.478369 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-home" pod "f05d1479-4fa1-4640-a6f5-59f43084b401" (UID: "f05d1479-4fa1-4640-a6f5-59f43084b401"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:47.478524 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:47.478479 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-dshm\") pod \"f05d1479-4fa1-4640-a6f5-59f43084b401\" (UID: \"f05d1479-4fa1-4640-a6f5-59f43084b401\") " Oct 30 21:35:47.478524 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:47.478470 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-tmp-dir" pod "f05d1479-4fa1-4640-a6f5-59f43084b401" (UID: "f05d1479-4fa1-4640-a6f5-59f43084b401"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:47.479424 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:47.479402 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:47.479516 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:47.479428 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:47.479516 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:47.479439 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:47.480197 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:47.479846 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f05d1479-4fa1-4640-a6f5-59f43084b401-tls-certs" pod "f05d1479-4fa1-4640-a6f5-59f43084b401" (UID: "f05d1479-4fa1-4640-a6f5-59f43084b401"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:35:47.480197 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:47.479963 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f05d1479-4fa1-4640-a6f5-59f43084b401-kube-api-access-kt7zm" pod "f05d1479-4fa1-4640-a6f5-59f43084b401" (UID: "f05d1479-4fa1-4640-a6f5-59f43084b401"). InnerVolumeSpecName "kube-api-access-kt7zm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:47.481495 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:47.481473 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-dshm" pod "f05d1479-4fa1-4640-a6f5-59f43084b401" (UID: "f05d1479-4fa1-4640-a6f5-59f43084b401"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:35:47.580766 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:47.580723 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f05d1479-4fa1-4640-a6f5-59f43084b401-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:47.580766 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:47.580755 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt7zm\" (UniqueName: \"kubernetes.io/projected/f05d1479-4fa1-4640-a6f5-59f43084b401-kube-api-access-kt7zm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:47.580766 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:47.580772 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f05d1479-4fa1-4640-a6f5-59f43084b401-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:48.314715 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:48.314683 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" Oct 30 21:35:48.315300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:48.314686 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp" event={"ID":"f05d1479-4fa1-4640-a6f5-59f43084b401","Type":"ContainerDied","Data":"c04a20ff1da9618a0d8e259b5e082129a8681a67d2377f283494fae68d50576f"} Oct 30 21:35:48.315300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:48.314818 4926 scope.go:122] "RemoveContainer" containerID="1d1cd4004e14bb6aac1c584def6c23b4ac39f1a6b37c27e7db0bc8fd137b4ccc" Oct 30 21:35:48.380115 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:48.380076 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp"] Oct 30 21:35:48.387811 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:48.387776 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-290ede/test-model-290ede-kserve-dbdd9f749-jvjdp"] Oct 30 21:35:49.018391 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:49.018353 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f05d1479-4fa1-4640-a6f5-59f43084b401" path="/var/lib/kubelet/pods/f05d1479-4fa1-4640-a6f5-59f43084b401/volumes" Oct 30 21:35:55.357759 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:55.357713 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2"] Oct 30 21:35:55.904389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:55.901319 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n"] Oct 30 21:35:55.904389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:55.902471 4926 cpu_manager.go:414] "containerMap: removing container" podUID="24d947c8-b5e0-4d7c-8517-c2fd7984424d" containerName="revoke-keys" Oct 30 21:35:55.904389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:55.902493 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24d947c8-b5e0-4d7c-8517-c2fd7984424d" containerName="revoke-keys" Oct 30 21:35:55.904389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:55.902514 4926 cpu_manager.go:414] "containerMap: removing container" podUID="f05d1479-4fa1-4640-a6f5-59f43084b401" containerName="main" Oct 30 21:35:55.904389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:55.902522 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f05d1479-4fa1-4640-a6f5-59f43084b401" containerName="main" Oct 30 21:35:55.904389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:55.902550 4926 cpu_manager.go:414] "containerMap: removing container" podUID="9cbcfa4c-ad1b-4110-a069-38c79809bc98" containerName="istio-proxy" Oct 30 21:35:55.904389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:55.902559 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cbcfa4c-ad1b-4110-a069-38c79809bc98" containerName="istio-proxy" Oct 30 21:35:55.904389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:55.902778 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="24d947c8-b5e0-4d7c-8517-c2fd7984424d" containerName="revoke-keys" Oct 30 21:35:55.904389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:55.902794 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="f05d1479-4fa1-4640-a6f5-59f43084b401" containerName="main" Oct 30 21:35:55.904389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:55.902806 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cbcfa4c-ad1b-4110-a069-38c79809bc98" containerName="istio-proxy" Oct 30 21:35:55.909319 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:55.909292 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n" Oct 30 21:35:55.929018 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:55.928984 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n"] Oct 30 21:35:55.934346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:55.934301 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:35:55.934523 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:55.934389 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:35:56.053563 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:56.053516 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a213e915-71a7-4c50-b0d4-58f1f5210028-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf8f82a0-pcz5n\" (UID: \"a213e915-71a7-4c50-b0d4-58f1f5210028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n" Oct 30 21:35:56.053769 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:56.053695 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a213e915-71a7-4c50-b0d4-58f1f5210028-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf8f82a0-pcz5n\" (UID: \"a213e915-71a7-4c50-b0d4-58f1f5210028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n" Oct 30 21:35:56.053769 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:56.053756 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mthv\" (UniqueName: \"kubernetes.io/projected/a213e915-71a7-4c50-b0d4-58f1f5210028-kube-api-access-4mthv\") pod \"maas-api-revoke-sub-cf8f82a0-pcz5n\" (UID: \"a213e915-71a7-4c50-b0d4-58f1f5210028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n" Oct 30 21:35:56.155562 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:56.155472 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a213e915-71a7-4c50-b0d4-58f1f5210028-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf8f82a0-pcz5n\" (UID: \"a213e915-71a7-4c50-b0d4-58f1f5210028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n" Oct 30 21:35:56.155810 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:56.155582 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a213e915-71a7-4c50-b0d4-58f1f5210028-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf8f82a0-pcz5n\" (UID: \"a213e915-71a7-4c50-b0d4-58f1f5210028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n" Oct 30 21:35:56.155810 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:56.155615 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mthv\" (UniqueName: \"kubernetes.io/projected/a213e915-71a7-4c50-b0d4-58f1f5210028-kube-api-access-4mthv\") pod \"maas-api-revoke-sub-cf8f82a0-pcz5n\" (UID: \"a213e915-71a7-4c50-b0d4-58f1f5210028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n" Oct 30 21:35:56.156423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:56.156396 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a213e915-71a7-4c50-b0d4-58f1f5210028-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf8f82a0-pcz5n\" (UID: \"a213e915-71a7-4c50-b0d4-58f1f5210028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n" Oct 30 21:35:56.157995 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:56.157969 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a213e915-71a7-4c50-b0d4-58f1f5210028-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf8f82a0-pcz5n\" (UID: \"a213e915-71a7-4c50-b0d4-58f1f5210028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n" Oct 30 21:35:56.178666 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:56.178630 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mthv\" (UniqueName: \"kubernetes.io/projected/a213e915-71a7-4c50-b0d4-58f1f5210028-kube-api-access-4mthv\") pod \"maas-api-revoke-sub-cf8f82a0-pcz5n\" (UID: \"a213e915-71a7-4c50-b0d4-58f1f5210028\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n" Oct 30 21:35:56.230515 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:56.230477 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n" Oct 30 21:35:56.575777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:56.575751 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n"] Oct 30 21:35:56.578607 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:35:56.578567 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda213e915_71a7_4c50_b0d4_58f1f5210028.slice/crio-3a135b40b1a1147cbd52937ad7aa52d057d309c4de9306e25c620972b2645140 WatchSource:0}: Error finding container 3a135b40b1a1147cbd52937ad7aa52d057d309c4de9306e25c620972b2645140: Status 404 returned error can't find the container with id 3a135b40b1a1147cbd52937ad7aa52d057d309c4de9306e25c620972b2645140 Oct 30 21:35:57.417742 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:57.417703 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n" event={"ID":"a213e915-71a7-4c50-b0d4-58f1f5210028","Type":"ContainerStarted","Data":"258dc55b949406425aa1582e498a0708577e5c6042633a97c5e51ea98a46b309"} Oct 30 21:35:57.417963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:57.417757 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n" event={"ID":"a213e915-71a7-4c50-b0d4-58f1f5210028","Type":"ContainerStarted","Data":"3a135b40b1a1147cbd52937ad7aa52d057d309c4de9306e25c620972b2645140"} Oct 30 21:35:57.464559 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:57.464502 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n" podStartSLOduration=2.464487518 podStartE2EDuration="2.464487518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:35:57.460478076 +0000 UTC m=+2144.581551816" watchObservedRunningTime="2026-10-30 21:35:57.464487518 +0000 UTC m=+2144.585561264" Oct 30 21:35:58.429118 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:58.429080 4926 generic.go:356] "Generic (PLEG): container finished" podID="a213e915-71a7-4c50-b0d4-58f1f5210028" containerID="258dc55b949406425aa1582e498a0708577e5c6042633a97c5e51ea98a46b309" exitCode=0 Oct 30 21:35:58.429526 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:58.429162 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n" event={"ID":"a213e915-71a7-4c50-b0d4-58f1f5210028","Type":"ContainerDied","Data":"258dc55b949406425aa1582e498a0708577e5c6042633a97c5e51ea98a46b309"} Oct 30 21:35:58.818874 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:58.818769 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22"] Oct 30 21:35:59.317735 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.317691 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds"] Oct 30 21:35:59.322443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.322418 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds" Oct 30 21:35:59.350493 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.350460 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds"] Oct 30 21:35:59.419271 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.419237 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9j8n\" (UniqueName: \"kubernetes.io/projected/cd8d58bc-798f-4e13-ad51-8eba733fb434-kube-api-access-c9j8n\") pod \"maas-api-revoke-sub-8ac19acb-ng7ds\" (UID: \"cd8d58bc-798f-4e13-ad51-8eba733fb434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds" Oct 30 21:35:59.419477 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.419452 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd8d58bc-798f-4e13-ad51-8eba733fb434-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ac19acb-ng7ds\" (UID: \"cd8d58bc-798f-4e13-ad51-8eba733fb434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds" Oct 30 21:35:59.419589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.419573 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd8d58bc-798f-4e13-ad51-8eba733fb434-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ac19acb-ng7ds\" (UID: \"cd8d58bc-798f-4e13-ad51-8eba733fb434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds" Oct 30 21:35:59.529755 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.529706 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9j8n\" (UniqueName: \"kubernetes.io/projected/cd8d58bc-798f-4e13-ad51-8eba733fb434-kube-api-access-c9j8n\") pod \"maas-api-revoke-sub-8ac19acb-ng7ds\" (UID: \"cd8d58bc-798f-4e13-ad51-8eba733fb434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds" Oct 30 21:35:59.530263 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.529940 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd8d58bc-798f-4e13-ad51-8eba733fb434-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ac19acb-ng7ds\" (UID: \"cd8d58bc-798f-4e13-ad51-8eba733fb434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds" Oct 30 21:35:59.530263 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.530090 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd8d58bc-798f-4e13-ad51-8eba733fb434-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ac19acb-ng7ds\" (UID: \"cd8d58bc-798f-4e13-ad51-8eba733fb434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds" Oct 30 21:35:59.531477 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.531427 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd8d58bc-798f-4e13-ad51-8eba733fb434-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ac19acb-ng7ds\" (UID: \"cd8d58bc-798f-4e13-ad51-8eba733fb434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds" Oct 30 21:35:59.536101 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.536012 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd8d58bc-798f-4e13-ad51-8eba733fb434-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ac19acb-ng7ds\" (UID: \"cd8d58bc-798f-4e13-ad51-8eba733fb434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds" Oct 30 21:35:59.557585 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.557552 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9j8n\" (UniqueName: \"kubernetes.io/projected/cd8d58bc-798f-4e13-ad51-8eba733fb434-kube-api-access-c9j8n\") pod \"maas-api-revoke-sub-8ac19acb-ng7ds\" (UID: \"cd8d58bc-798f-4e13-ad51-8eba733fb434\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds" Oct 30 21:35:59.623674 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.623644 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n" Oct 30 21:35:59.637961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.637927 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds" Oct 30 21:35:59.733280 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.733247 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a213e915-71a7-4c50-b0d4-58f1f5210028-kube-api-access-4mthv\" (UniqueName: \"kubernetes.io/projected/a213e915-71a7-4c50-b0d4-58f1f5210028-kube-api-access-4mthv\") pod \"a213e915-71a7-4c50-b0d4-58f1f5210028\" (UID: \"a213e915-71a7-4c50-b0d4-58f1f5210028\") " Oct 30 21:35:59.733448 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.733389 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a213e915-71a7-4c50-b0d4-58f1f5210028-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a213e915-71a7-4c50-b0d4-58f1f5210028-maas-api-xks-ca\") pod \"a213e915-71a7-4c50-b0d4-58f1f5210028\" (UID: \"a213e915-71a7-4c50-b0d4-58f1f5210028\") " Oct 30 21:35:59.734460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.733966 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a213e915-71a7-4c50-b0d4-58f1f5210028-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a213e915-71a7-4c50-b0d4-58f1f5210028-maas-api-openshift-ca\") pod \"a213e915-71a7-4c50-b0d4-58f1f5210028\" (UID: \"a213e915-71a7-4c50-b0d4-58f1f5210028\") " Oct 30 21:35:59.735202 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.735165 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a213e915-71a7-4c50-b0d4-58f1f5210028-maas-api-openshift-ca" pod "a213e915-71a7-4c50-b0d4-58f1f5210028" (UID: "a213e915-71a7-4c50-b0d4-58f1f5210028"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:35:59.736659 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.736617 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a213e915-71a7-4c50-b0d4-58f1f5210028-kube-api-access-4mthv" pod "a213e915-71a7-4c50-b0d4-58f1f5210028" (UID: "a213e915-71a7-4c50-b0d4-58f1f5210028"). InnerVolumeSpecName "kube-api-access-4mthv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:35:59.738621 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.738591 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a213e915-71a7-4c50-b0d4-58f1f5210028-maas-api-xks-ca" pod "a213e915-71a7-4c50-b0d4-58f1f5210028" (UID: "a213e915-71a7-4c50-b0d4-58f1f5210028"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:35:59.836165 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.836128 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mthv\" (UniqueName: \"kubernetes.io/projected/a213e915-71a7-4c50-b0d4-58f1f5210028-kube-api-access-4mthv\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:59.836165 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.836163 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a213e915-71a7-4c50-b0d4-58f1f5210028-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:59.836165 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.836175 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a213e915-71a7-4c50-b0d4-58f1f5210028-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:35:59.967934 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:35:59.967901 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds"] Oct 30 21:35:59.970708 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:35:59.970645 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8d58bc_798f_4e13_ad51_8eba733fb434.slice/crio-3841039badff81814817cc4cc4160b87a611d45f71cfac65e482bbbac1aeb693 WatchSource:0}: Error finding container 3841039badff81814817cc4cc4160b87a611d45f71cfac65e482bbbac1aeb693: Status 404 returned error can't find the container with id 3841039badff81814817cc4cc4160b87a611d45f71cfac65e482bbbac1aeb693 Oct 30 21:36:00.448173 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:00.448142 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n" Oct 30 21:36:00.448173 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:00.448152 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n" event={"ID":"a213e915-71a7-4c50-b0d4-58f1f5210028","Type":"ContainerDied","Data":"3a135b40b1a1147cbd52937ad7aa52d057d309c4de9306e25c620972b2645140"} Oct 30 21:36:00.448419 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:00.448209 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a135b40b1a1147cbd52937ad7aa52d057d309c4de9306e25c620972b2645140" Oct 30 21:36:00.450998 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:00.450407 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds" event={"ID":"cd8d58bc-798f-4e13-ad51-8eba733fb434","Type":"ContainerStarted","Data":"f49becfd602a9db4615b1084eb4e390ac55e87a33a5ca7ce5397258fea07955e"} Oct 30 21:36:00.450998 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:00.450464 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds" event={"ID":"cd8d58bc-798f-4e13-ad51-8eba733fb434","Type":"ContainerStarted","Data":"3841039badff81814817cc4cc4160b87a611d45f71cfac65e482bbbac1aeb693"} Oct 30 21:36:00.492438 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:00.492381 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds" podStartSLOduration=1.49236461 podStartE2EDuration="1.49236461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:00.488265578 +0000 UTC m=+2147.609339325" watchObservedRunningTime="2026-10-30 21:36:00.49236461 +0000 UTC m=+2147.613438404" Oct 30 21:36:01.460384 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:01.460346 4926 generic.go:356] "Generic (PLEG): container finished" podID="cd8d58bc-798f-4e13-ad51-8eba733fb434" containerID="f49becfd602a9db4615b1084eb4e390ac55e87a33a5ca7ce5397258fea07955e" exitCode=0 Oct 30 21:36:01.460968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:01.460438 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds" event={"ID":"cd8d58bc-798f-4e13-ad51-8eba733fb434","Type":"ContainerDied","Data":"f49becfd602a9db4615b1084eb4e390ac55e87a33a5ca7ce5397258fea07955e"} Oct 30 21:36:02.659157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:02.659124 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds" Oct 30 21:36:02.782545 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:02.782448 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd8d58bc-798f-4e13-ad51-8eba733fb434-kube-api-access-c9j8n\" (UniqueName: \"kubernetes.io/projected/cd8d58bc-798f-4e13-ad51-8eba733fb434-kube-api-access-c9j8n\") pod \"cd8d58bc-798f-4e13-ad51-8eba733fb434\" (UID: \"cd8d58bc-798f-4e13-ad51-8eba733fb434\") " Oct 30 21:36:02.782738 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:02.782564 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd8d58bc-798f-4e13-ad51-8eba733fb434-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd8d58bc-798f-4e13-ad51-8eba733fb434-maas-api-xks-ca\") pod \"cd8d58bc-798f-4e13-ad51-8eba733fb434\" (UID: \"cd8d58bc-798f-4e13-ad51-8eba733fb434\") " Oct 30 21:36:02.783108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:02.783064 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd8d58bc-798f-4e13-ad51-8eba733fb434-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd8d58bc-798f-4e13-ad51-8eba733fb434-maas-api-openshift-ca\") pod \"cd8d58bc-798f-4e13-ad51-8eba733fb434\" (UID: \"cd8d58bc-798f-4e13-ad51-8eba733fb434\") " Oct 30 21:36:02.783439 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:02.783410 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd8d58bc-798f-4e13-ad51-8eba733fb434-maas-api-openshift-ca" pod "cd8d58bc-798f-4e13-ad51-8eba733fb434" (UID: "cd8d58bc-798f-4e13-ad51-8eba733fb434"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:02.783807 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:02.783787 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd8d58bc-798f-4e13-ad51-8eba733fb434-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:02.785623 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:02.785597 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd8d58bc-798f-4e13-ad51-8eba733fb434-maas-api-xks-ca" pod "cd8d58bc-798f-4e13-ad51-8eba733fb434" (UID: "cd8d58bc-798f-4e13-ad51-8eba733fb434"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:36:02.785721 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:02.785648 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd8d58bc-798f-4e13-ad51-8eba733fb434-kube-api-access-c9j8n" pod "cd8d58bc-798f-4e13-ad51-8eba733fb434" (UID: "cd8d58bc-798f-4e13-ad51-8eba733fb434"). InnerVolumeSpecName "kube-api-access-c9j8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:02.884953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:02.884904 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd8d58bc-798f-4e13-ad51-8eba733fb434-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:02.884953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:02.884944 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9j8n\" (UniqueName: \"kubernetes.io/projected/cd8d58bc-798f-4e13-ad51-8eba733fb434-kube-api-access-c9j8n\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:03.482711 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:03.482664 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds" event={"ID":"cd8d58bc-798f-4e13-ad51-8eba733fb434","Type":"ContainerDied","Data":"3841039badff81814817cc4cc4160b87a611d45f71cfac65e482bbbac1aeb693"} Oct 30 21:36:03.482711 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:03.482690 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds" Oct 30 21:36:03.482994 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:03.482731 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3841039badff81814817cc4cc4160b87a611d45f71cfac65e482bbbac1aeb693" Oct 30 21:36:05.204473 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:05.204429 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh"] Oct 30 21:36:05.205159 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:05.204703 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" podUID="df81fda1-3052-4fb9-b9aa-5674a2e5847a" containerName="istio-proxy" containerID="cri-o://097071ca8e288930d0fa88637ea631ebd5670fffe267c39222875736edc65c33" gracePeriod=30 Oct 30 21:36:07.490890 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:36:07.490845 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:36:09.049762 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:09.049716 4926 patch_prober.go:29] interesting pod/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:36:09.050350 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:09.049784 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" podUID="df81fda1-3052-4fb9-b9aa-5674a2e5847a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:36:10.391904 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.391381 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" podUID="2ae29b4c-757d-49a0-9dd0-fa0999ab01c4" containerName="main" containerID="cri-o://5c39031f7254c4d641e8fbf20d67f61a6385fb260eea6a61c831f87b9cd521d6" gracePeriod=45 Oct 30 21:36:10.534923 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.534888 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:36:10.561397 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.561362 4926 generic.go:356] "Generic (PLEG): container finished" podID="df81fda1-3052-4fb9-b9aa-5674a2e5847a" containerID="097071ca8e288930d0fa88637ea631ebd5670fffe267c39222875736edc65c33" exitCode=0 Oct 30 21:36:10.561628 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.561608 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" Oct 30 21:36:10.561971 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.561545 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" event={"ID":"df81fda1-3052-4fb9-b9aa-5674a2e5847a","Type":"ContainerDied","Data":"097071ca8e288930d0fa88637ea631ebd5670fffe267c39222875736edc65c33"} Oct 30 21:36:10.562140 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.562122 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh" event={"ID":"df81fda1-3052-4fb9-b9aa-5674a2e5847a","Type":"ContainerDied","Data":"6cf8442391fceed5352c3a2915dc3c3c329e8e623d353480003229c21d293861"} Oct 30 21:36:10.562255 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.562242 4926 scope.go:122] "RemoveContainer" containerID="097071ca8e288930d0fa88637ea631ebd5670fffe267c39222875736edc65c33" Oct 30 21:36:10.564268 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.564249 4926 generic.go:356] "Generic (PLEG): container finished" podID="2ae29b4c-757d-49a0-9dd0-fa0999ab01c4" containerID="5c39031f7254c4d641e8fbf20d67f61a6385fb260eea6a61c831f87b9cd521d6" exitCode=0 Oct 30 21:36:10.564355 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.564319 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" event={"ID":"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4","Type":"ContainerDied","Data":"5c39031f7254c4d641e8fbf20d67f61a6385fb260eea6a61c831f87b9cd521d6"} Oct 30 21:36:10.576861 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.576842 4926 scope.go:122] "RemoveContainer" containerID="097071ca8e288930d0fa88637ea631ebd5670fffe267c39222875736edc65c33" Oct 30 21:36:10.577359 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:36:10.577333 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"097071ca8e288930d0fa88637ea631ebd5670fffe267c39222875736edc65c33\": container with ID starting with 097071ca8e288930d0fa88637ea631ebd5670fffe267c39222875736edc65c33 not found: ID does not exist" containerID="097071ca8e288930d0fa88637ea631ebd5670fffe267c39222875736edc65c33" Oct 30 21:36:10.577469 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.577373 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"097071ca8e288930d0fa88637ea631ebd5670fffe267c39222875736edc65c33"} err="failed to get container status \"097071ca8e288930d0fa88637ea631ebd5670fffe267c39222875736edc65c33\": rpc error: code = NotFound desc = could not find container \"097071ca8e288930d0fa88637ea631ebd5670fffe267c39222875736edc65c33\": container with ID starting with 097071ca8e288930d0fa88637ea631ebd5670fffe267c39222875736edc65c33 not found: ID does not exist" Oct 30 21:36:10.614058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.612420 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-data\") pod \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " Oct 30 21:36:10.614058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.612475 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-envoy\") pod \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " Oct 30 21:36:10.614058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.612521 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-credential-socket\") pod \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " Oct 30 21:36:10.614058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.612566 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-token\" (UniqueName: \"kubernetes.io/projected/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-token\") pod \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " Oct 30 21:36:10.614058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.612637 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-workload-certs\") pod \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " Oct 30 21:36:10.614058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.612793 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df81fda1-3052-4fb9-b9aa-5674a2e5847a-kube-api-access-std2t\" (UniqueName: \"kubernetes.io/projected/df81fda1-3052-4fb9-b9aa-5674a2e5847a-kube-api-access-std2t\") pod \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " Oct 30 21:36:10.614058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.612897 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-workload-socket\") pod \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " Oct 30 21:36:10.614058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.612957 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-podinfo\") pod \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " Oct 30 21:36:10.614058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.613095 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istiod-ca-cert\") pod \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\" (UID: \"df81fda1-3052-4fb9-b9aa-5674a2e5847a\") " Oct 30 21:36:10.614753 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.614282 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istiod-ca-cert" pod "df81fda1-3052-4fb9-b9aa-5674a2e5847a" (UID: "df81fda1-3052-4fb9-b9aa-5674a2e5847a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:10.614753 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.614616 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-workload-certs" pod "df81fda1-3052-4fb9-b9aa-5674a2e5847a" (UID: "df81fda1-3052-4fb9-b9aa-5674a2e5847a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:10.619159 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.619131 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-token" pod "df81fda1-3052-4fb9-b9aa-5674a2e5847a" (UID: "df81fda1-3052-4fb9-b9aa-5674a2e5847a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:10.619317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.619294 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-credential-socket" pod "df81fda1-3052-4fb9-b9aa-5674a2e5847a" (UID: "df81fda1-3052-4fb9-b9aa-5674a2e5847a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:10.619443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.619428 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-workload-socket" pod "df81fda1-3052-4fb9-b9aa-5674a2e5847a" (UID: "df81fda1-3052-4fb9-b9aa-5674a2e5847a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:10.619563 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.619551 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-data" pod "df81fda1-3052-4fb9-b9aa-5674a2e5847a" (UID: "df81fda1-3052-4fb9-b9aa-5674a2e5847a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:10.620673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.620647 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df81fda1-3052-4fb9-b9aa-5674a2e5847a-kube-api-access-std2t" pod "df81fda1-3052-4fb9-b9aa-5674a2e5847a" (UID: "df81fda1-3052-4fb9-b9aa-5674a2e5847a"). InnerVolumeSpecName "kube-api-access-std2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:10.623096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.623064 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-podinfo" pod "df81fda1-3052-4fb9-b9aa-5674a2e5847a" (UID: "df81fda1-3052-4fb9-b9aa-5674a2e5847a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:36:10.623885 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.623864 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-envoy" pod "df81fda1-3052-4fb9-b9aa-5674a2e5847a" (UID: "df81fda1-3052-4fb9-b9aa-5674a2e5847a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:10.689687 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.689657 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:36:10.715747 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.715711 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:10.715747 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.715738 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:10.715747 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.715746 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:10.715747 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.715755 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:10.716177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.715765 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:10.716177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.715775 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:10.716177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.715783 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-std2t\" (UniqueName: \"kubernetes.io/projected/df81fda1-3052-4fb9-b9aa-5674a2e5847a-kube-api-access-std2t\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:10.716177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.715792 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df81fda1-3052-4fb9-b9aa-5674a2e5847a-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:10.716177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.715800 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df81fda1-3052-4fb9-b9aa-5674a2e5847a-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:10.817305 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.817208 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-tls-certs\") pod \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " Oct 30 21:36:10.817467 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.817357 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-kube-api-access-kr6bb\" (UniqueName: \"kubernetes.io/projected/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-kube-api-access-kr6bb\") pod \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " Oct 30 21:36:10.817528 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.817462 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-model-cache\") pod \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " Oct 30 21:36:10.817582 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.817554 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-tmp-dir\") pod \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " Oct 30 21:36:10.817649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.817591 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-home\" (UniqueName: \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-home\") pod \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " Oct 30 21:36:10.817709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.817685 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-dshm\") pod \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\" (UID: \"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4\") " Oct 30 21:36:10.817801 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.817767 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-model-cache" pod "2ae29b4c-757d-49a0-9dd0-fa0999ab01c4" (UID: "2ae29b4c-757d-49a0-9dd0-fa0999ab01c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:10.818292 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.817865 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-tmp-dir" pod "2ae29b4c-757d-49a0-9dd0-fa0999ab01c4" (UID: "2ae29b4c-757d-49a0-9dd0-fa0999ab01c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:10.818292 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.817933 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-home" pod "2ae29b4c-757d-49a0-9dd0-fa0999ab01c4" (UID: "2ae29b4c-757d-49a0-9dd0-fa0999ab01c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:10.818511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.818445 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:10.818511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.818468 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:10.818511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.818482 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:10.819642 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.819619 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-tls-certs" pod "2ae29b4c-757d-49a0-9dd0-fa0999ab01c4" (UID: "2ae29b4c-757d-49a0-9dd0-fa0999ab01c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:36:10.819764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.819664 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-kube-api-access-kr6bb" pod "2ae29b4c-757d-49a0-9dd0-fa0999ab01c4" (UID: "2ae29b4c-757d-49a0-9dd0-fa0999ab01c4"). InnerVolumeSpecName "kube-api-access-kr6bb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:10.819861 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.819845 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-dshm" pod "2ae29b4c-757d-49a0-9dd0-fa0999ab01c4" (UID: "2ae29b4c-757d-49a0-9dd0-fa0999ab01c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:10.911469 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.911381 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh"] Oct 30 21:36:10.919842 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.919806 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:10.919842 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.919838 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:10.919842 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.919848 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr6bb\" (UniqueName: \"kubernetes.io/projected/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4-kube-api-access-kr6bb\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:10.921330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:10.921300 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-43c070-openshift-default-857c6bc8df-hdwzh"] Oct 30 21:36:11.018369 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:11.018332 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df81fda1-3052-4fb9-b9aa-5674a2e5847a" path="/var/lib/kubelet/pods/df81fda1-3052-4fb9-b9aa-5674a2e5847a/volumes" Oct 30 21:36:11.576420 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:11.576339 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" Oct 30 21:36:11.576897 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:11.576341 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" event={"ID":"2ae29b4c-757d-49a0-9dd0-fa0999ab01c4","Type":"ContainerDied","Data":"0de5a0fcd025f10d5242fde4f2a3e1206f3f82e550ae2dd8f15bed7183dbf848"} Oct 30 21:36:11.576897 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:11.576479 4926 scope.go:122] "RemoveContainer" containerID="5c39031f7254c4d641e8fbf20d67f61a6385fb260eea6a61c831f87b9cd521d6" Oct 30 21:36:11.623200 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:11.623153 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2" podUID="2ae29b4c-757d-49a0-9dd0-fa0999ab01c4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.176:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:36:11.630322 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:11.630285 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2"] Oct 30 21:36:11.634736 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:11.634704 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c5145c/test-model-c5145c-kserve-5fd549-697m2"] Oct 30 21:36:13.028843 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:13.028803 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae29b4c-757d-49a0-9dd0-fa0999ab01c4" path="/var/lib/kubelet/pods/2ae29b4c-757d-49a0-9dd0-fa0999ab01c4/volumes" Oct 30 21:36:13.362347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:13.362256 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:36:13.850185 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:13.850140 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" podUID="50ced360-7983-4b5a-8b03-35714cd87a9c" containerName="main" containerID="cri-o://bfdc50939933712eeb4cda35bb361990b417b01c89eb3c7cf222fc3adffb42ad" gracePeriod=45 Oct 30 21:36:13.949111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:13.949074 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" podUID="50ced360-7983-4b5a-8b03-35714cd87a9c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.159:8000/ready\": dial tcp 10.132.0.159:8000: connect: connection refused" Oct 30 21:36:14.150366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.150331 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:36:14.281910 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.281873 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-model-cache\") pod \"50ced360-7983-4b5a-8b03-35714cd87a9c\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " Oct 30 21:36:14.282151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.281949 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-dshm\") pod \"50ced360-7983-4b5a-8b03-35714cd87a9c\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " Oct 30 21:36:14.282151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.281970 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50ced360-7983-4b5a-8b03-35714cd87a9c-tls-certs\" (UniqueName: \"kubernetes.io/secret/50ced360-7983-4b5a-8b03-35714cd87a9c-tls-certs\") pod \"50ced360-7983-4b5a-8b03-35714cd87a9c\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " Oct 30 21:36:14.282297 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.282214 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-tmp-dir\") pod \"50ced360-7983-4b5a-8b03-35714cd87a9c\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " Oct 30 21:36:14.282297 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.282254 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-model-cache" pod "50ced360-7983-4b5a-8b03-35714cd87a9c" (UID: "50ced360-7983-4b5a-8b03-35714cd87a9c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:14.282411 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.282331 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-home\" (UniqueName: \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-home\") pod \"50ced360-7983-4b5a-8b03-35714cd87a9c\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " Oct 30 21:36:14.282411 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.282386 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50ced360-7983-4b5a-8b03-35714cd87a9c-kube-api-access-khmjd\" (UniqueName: \"kubernetes.io/projected/50ced360-7983-4b5a-8b03-35714cd87a9c-kube-api-access-khmjd\") pod \"50ced360-7983-4b5a-8b03-35714cd87a9c\" (UID: \"50ced360-7983-4b5a-8b03-35714cd87a9c\") " Oct 30 21:36:14.282539 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.282453 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-tmp-dir" pod "50ced360-7983-4b5a-8b03-35714cd87a9c" (UID: "50ced360-7983-4b5a-8b03-35714cd87a9c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:14.282654 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.282630 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-home" pod "50ced360-7983-4b5a-8b03-35714cd87a9c" (UID: "50ced360-7983-4b5a-8b03-35714cd87a9c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:14.283011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.282973 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:14.283011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.282999 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:14.283232 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.283045 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:14.284704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.284668 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50ced360-7983-4b5a-8b03-35714cd87a9c-tls-certs" pod "50ced360-7983-4b5a-8b03-35714cd87a9c" (UID: "50ced360-7983-4b5a-8b03-35714cd87a9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:36:14.284886 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.284868 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-dshm" pod "50ced360-7983-4b5a-8b03-35714cd87a9c" (UID: "50ced360-7983-4b5a-8b03-35714cd87a9c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:14.284957 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.284865 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ced360-7983-4b5a-8b03-35714cd87a9c-kube-api-access-khmjd" pod "50ced360-7983-4b5a-8b03-35714cd87a9c" (UID: "50ced360-7983-4b5a-8b03-35714cd87a9c"). InnerVolumeSpecName "kube-api-access-khmjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:14.384296 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.384197 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khmjd\" (UniqueName: \"kubernetes.io/projected/50ced360-7983-4b5a-8b03-35714cd87a9c-kube-api-access-khmjd\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:14.384296 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.384231 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50ced360-7983-4b5a-8b03-35714cd87a9c-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:14.384296 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.384240 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50ced360-7983-4b5a-8b03-35714cd87a9c-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:14.610194 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.610155 4926 generic.go:356] "Generic (PLEG): container finished" podID="50ced360-7983-4b5a-8b03-35714cd87a9c" containerID="bfdc50939933712eeb4cda35bb361990b417b01c89eb3c7cf222fc3adffb42ad" exitCode=0 Oct 30 21:36:14.610392 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.610198 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" Oct 30 21:36:14.610392 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.610242 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" event={"ID":"50ced360-7983-4b5a-8b03-35714cd87a9c","Type":"ContainerDied","Data":"bfdc50939933712eeb4cda35bb361990b417b01c89eb3c7cf222fc3adffb42ad"} Oct 30 21:36:14.610392 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.610285 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22" event={"ID":"50ced360-7983-4b5a-8b03-35714cd87a9c","Type":"ContainerDied","Data":"567fbc0028298e65749ddab15cc828733242f5b897cb0218b0a2d74b00290b9a"} Oct 30 21:36:14.610392 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.610306 4926 scope.go:122] "RemoveContainer" containerID="bfdc50939933712eeb4cda35bb361990b417b01c89eb3c7cf222fc3adffb42ad" Oct 30 21:36:14.623374 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.623350 4926 scope.go:122] "RemoveContainer" containerID="bfdc50939933712eeb4cda35bb361990b417b01c89eb3c7cf222fc3adffb42ad" Oct 30 21:36:14.623721 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:36:14.623695 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfdc50939933712eeb4cda35bb361990b417b01c89eb3c7cf222fc3adffb42ad\": container with ID starting with bfdc50939933712eeb4cda35bb361990b417b01c89eb3c7cf222fc3adffb42ad not found: ID does not exist" containerID="bfdc50939933712eeb4cda35bb361990b417b01c89eb3c7cf222fc3adffb42ad" Oct 30 21:36:14.623815 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.623730 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfdc50939933712eeb4cda35bb361990b417b01c89eb3c7cf222fc3adffb42ad"} err="failed to get container status \"bfdc50939933712eeb4cda35bb361990b417b01c89eb3c7cf222fc3adffb42ad\": rpc error: code = NotFound desc = could not find container \"bfdc50939933712eeb4cda35bb361990b417b01c89eb3c7cf222fc3adffb42ad\": container with ID starting with bfdc50939933712eeb4cda35bb361990b417b01c89eb3c7cf222fc3adffb42ad not found: ID does not exist" Oct 30 21:36:14.673433 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.673395 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22"] Oct 30 21:36:14.685440 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:14.685405 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-79c24a/test-model-79c24a-kserve-7bf88d7d67-clv22"] Oct 30 21:36:15.053294 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.053209 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50ced360-7983-4b5a-8b03-35714cd87a9c" path="/var/lib/kubelet/pods/50ced360-7983-4b5a-8b03-35714cd87a9c/volumes" Oct 30 21:36:15.074068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.068417 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n"] Oct 30 21:36:15.074068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.069493 4926 cpu_manager.go:414] "containerMap: removing container" podUID="df81fda1-3052-4fb9-b9aa-5674a2e5847a" containerName="istio-proxy" Oct 30 21:36:15.074068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.069519 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df81fda1-3052-4fb9-b9aa-5674a2e5847a" containerName="istio-proxy" Oct 30 21:36:15.074068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.069568 4926 cpu_manager.go:414] "containerMap: removing container" podUID="a213e915-71a7-4c50-b0d4-58f1f5210028" containerName="revoke-keys" Oct 30 21:36:15.074068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.069576 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a213e915-71a7-4c50-b0d4-58f1f5210028" containerName="revoke-keys" Oct 30 21:36:15.074068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.069604 4926 cpu_manager.go:414] "containerMap: removing container" podUID="2ae29b4c-757d-49a0-9dd0-fa0999ab01c4" containerName="main" Oct 30 21:36:15.074068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.069614 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae29b4c-757d-49a0-9dd0-fa0999ab01c4" containerName="main" Oct 30 21:36:15.074068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.069631 4926 cpu_manager.go:414] "containerMap: removing container" podUID="cd8d58bc-798f-4e13-ad51-8eba733fb434" containerName="revoke-keys" Oct 30 21:36:15.074068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.069638 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd8d58bc-798f-4e13-ad51-8eba733fb434" containerName="revoke-keys" Oct 30 21:36:15.074068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.069671 4926 cpu_manager.go:414] "containerMap: removing container" podUID="50ced360-7983-4b5a-8b03-35714cd87a9c" containerName="main" Oct 30 21:36:15.074068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.069681 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ced360-7983-4b5a-8b03-35714cd87a9c" containerName="main" Oct 30 21:36:15.074068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.069816 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="50ced360-7983-4b5a-8b03-35714cd87a9c" containerName="main" Oct 30 21:36:15.074068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.069828 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="df81fda1-3052-4fb9-b9aa-5674a2e5847a" containerName="istio-proxy" Oct 30 21:36:15.074068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.069843 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="a213e915-71a7-4c50-b0d4-58f1f5210028" containerName="revoke-keys" Oct 30 21:36:15.074068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.069855 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae29b4c-757d-49a0-9dd0-fa0999ab01c4" containerName="main" Oct 30 21:36:15.074068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.069863 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd8d58bc-798f-4e13-ad51-8eba733fb434" containerName="revoke-keys" Oct 30 21:36:15.075719 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.075692 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n" Oct 30 21:36:15.125185 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.115063 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:36:15.126460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.125454 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:36:15.134427 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.134382 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n"] Oct 30 21:36:15.210981 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.208997 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04df2344-f9e9-4121-887e-0915c6714f8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dd5ed0f-s2d8n\" (UID: \"04df2344-f9e9-4121-887e-0915c6714f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n" Oct 30 21:36:15.210981 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.209240 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04df2344-f9e9-4121-887e-0915c6714f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dd5ed0f-s2d8n\" (UID: \"04df2344-f9e9-4121-887e-0915c6714f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n" Oct 30 21:36:15.210981 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.209459 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp8qx\" (UniqueName: \"kubernetes.io/projected/04df2344-f9e9-4121-887e-0915c6714f8c-kube-api-access-rp8qx\") pod \"maas-api-revoke-sub-2dd5ed0f-s2d8n\" (UID: \"04df2344-f9e9-4121-887e-0915c6714f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n" Oct 30 21:36:15.313138 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.310951 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp8qx\" (UniqueName: \"kubernetes.io/projected/04df2344-f9e9-4121-887e-0915c6714f8c-kube-api-access-rp8qx\") pod \"maas-api-revoke-sub-2dd5ed0f-s2d8n\" (UID: \"04df2344-f9e9-4121-887e-0915c6714f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n" Oct 30 21:36:15.313138 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.311244 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04df2344-f9e9-4121-887e-0915c6714f8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dd5ed0f-s2d8n\" (UID: \"04df2344-f9e9-4121-887e-0915c6714f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n" Oct 30 21:36:15.313138 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.311441 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04df2344-f9e9-4121-887e-0915c6714f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dd5ed0f-s2d8n\" (UID: \"04df2344-f9e9-4121-887e-0915c6714f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n" Oct 30 21:36:15.313138 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.312455 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04df2344-f9e9-4121-887e-0915c6714f8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dd5ed0f-s2d8n\" (UID: \"04df2344-f9e9-4121-887e-0915c6714f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n" Oct 30 21:36:15.335070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.332969 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04df2344-f9e9-4121-887e-0915c6714f8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dd5ed0f-s2d8n\" (UID: \"04df2344-f9e9-4121-887e-0915c6714f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n" Oct 30 21:36:15.435305 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.420818 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp8qx\" (UniqueName: \"kubernetes.io/projected/04df2344-f9e9-4121-887e-0915c6714f8c-kube-api-access-rp8qx\") pod \"maas-api-revoke-sub-2dd5ed0f-s2d8n\" (UID: \"04df2344-f9e9-4121-887e-0915c6714f8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n" Oct 30 21:36:15.435305 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:15.429973 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n" Oct 30 21:36:16.163622 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:16.163495 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp"] Oct 30 21:36:16.437870 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:16.437834 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n"] Oct 30 21:36:16.531077 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:16.530319 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq"] Oct 30 21:36:16.539657 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:16.539554 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq" Oct 30 21:36:16.556488 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:16.556452 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq"] Oct 30 21:36:16.637331 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:16.637285 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75zst\" (UniqueName: \"kubernetes.io/projected/d6ef610a-9e6f-47cc-9e18-7406909d5df1-kube-api-access-75zst\") pod \"maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq\" (UID: \"d6ef610a-9e6f-47cc-9e18-7406909d5df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq" Oct 30 21:36:16.637595 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:16.637574 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6ef610a-9e6f-47cc-9e18-7406909d5df1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq\" (UID: \"d6ef610a-9e6f-47cc-9e18-7406909d5df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq" Oct 30 21:36:16.637673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:16.637614 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6ef610a-9e6f-47cc-9e18-7406909d5df1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq\" (UID: \"d6ef610a-9e6f-47cc-9e18-7406909d5df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq" Oct 30 21:36:16.680129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:16.672067 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n" event={"ID":"04df2344-f9e9-4121-887e-0915c6714f8c","Type":"ContainerStarted","Data":"72976ac8df23c6692ece5a3e92104e84629086ae93c04eab75926db86a7911b1"} Oct 30 21:36:16.752435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:16.752381 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6ef610a-9e6f-47cc-9e18-7406909d5df1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq\" (UID: \"d6ef610a-9e6f-47cc-9e18-7406909d5df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq" Oct 30 21:36:16.752648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:16.752443 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6ef610a-9e6f-47cc-9e18-7406909d5df1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq\" (UID: \"d6ef610a-9e6f-47cc-9e18-7406909d5df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq" Oct 30 21:36:16.752648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:16.752576 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75zst\" (UniqueName: \"kubernetes.io/projected/d6ef610a-9e6f-47cc-9e18-7406909d5df1-kube-api-access-75zst\") pod \"maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq\" (UID: \"d6ef610a-9e6f-47cc-9e18-7406909d5df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq" Oct 30 21:36:16.756311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:16.754311 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6ef610a-9e6f-47cc-9e18-7406909d5df1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq\" (UID: \"d6ef610a-9e6f-47cc-9e18-7406909d5df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq" Oct 30 21:36:16.795887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:16.795860 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6ef610a-9e6f-47cc-9e18-7406909d5df1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq\" (UID: \"d6ef610a-9e6f-47cc-9e18-7406909d5df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq" Oct 30 21:36:16.828122 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:16.827637 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75zst\" (UniqueName: \"kubernetes.io/projected/d6ef610a-9e6f-47cc-9e18-7406909d5df1-kube-api-access-75zst\") pod \"maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq\" (UID: \"d6ef610a-9e6f-47cc-9e18-7406909d5df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq" Oct 30 21:36:16.876021 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:16.875981 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6"] Oct 30 21:36:16.880717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:16.880681 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6" Oct 30 21:36:16.903409 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:16.903368 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6"] Oct 30 21:36:16.927405 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:16.927369 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq" Oct 30 21:36:16.963428 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:16.963382 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28246ca0-71ac-4f01-bc42-52b5afbedc3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6\" (UID: \"28246ca0-71ac-4f01-bc42-52b5afbedc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6" Oct 30 21:36:16.963626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:16.963478 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwc7g\" (UniqueName: \"kubernetes.io/projected/28246ca0-71ac-4f01-bc42-52b5afbedc3a-kube-api-access-zwc7g\") pod \"maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6\" (UID: \"28246ca0-71ac-4f01-bc42-52b5afbedc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6" Oct 30 21:36:16.963781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:16.963756 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28246ca0-71ac-4f01-bc42-52b5afbedc3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6\" (UID: \"28246ca0-71ac-4f01-bc42-52b5afbedc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6" Oct 30 21:36:17.065375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:17.065328 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28246ca0-71ac-4f01-bc42-52b5afbedc3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6\" (UID: \"28246ca0-71ac-4f01-bc42-52b5afbedc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6" Oct 30 21:36:17.065624 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:17.065596 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwc7g\" (UniqueName: \"kubernetes.io/projected/28246ca0-71ac-4f01-bc42-52b5afbedc3a-kube-api-access-zwc7g\") pod \"maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6\" (UID: \"28246ca0-71ac-4f01-bc42-52b5afbedc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6" Oct 30 21:36:17.065888 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:17.065869 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28246ca0-71ac-4f01-bc42-52b5afbedc3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6\" (UID: \"28246ca0-71ac-4f01-bc42-52b5afbedc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6" Oct 30 21:36:17.066696 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:17.066669 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28246ca0-71ac-4f01-bc42-52b5afbedc3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6\" (UID: \"28246ca0-71ac-4f01-bc42-52b5afbedc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6" Oct 30 21:36:17.068206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:17.068183 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28246ca0-71ac-4f01-bc42-52b5afbedc3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6\" (UID: \"28246ca0-71ac-4f01-bc42-52b5afbedc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6" Oct 30 21:36:17.092557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:17.092527 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwc7g\" (UniqueName: \"kubernetes.io/projected/28246ca0-71ac-4f01-bc42-52b5afbedc3a-kube-api-access-zwc7g\") pod \"maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6\" (UID: \"28246ca0-71ac-4f01-bc42-52b5afbedc3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6" Oct 30 21:36:17.208374 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:17.208287 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6" Oct 30 21:36:17.270930 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:17.270892 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq"] Oct 30 21:36:17.274515 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:36:17.274486 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6ef610a_9e6f_47cc_9e18_7406909d5df1.slice/crio-7f7d9453f19bfbd92cbd7b1e3b8eacb0e0bcc6466f867077da49359b3d3cf5bf WatchSource:0}: Error finding container 7f7d9453f19bfbd92cbd7b1e3b8eacb0e0bcc6466f867077da49359b3d3cf5bf: Status 404 returned error can't find the container with id 7f7d9453f19bfbd92cbd7b1e3b8eacb0e0bcc6466f867077da49359b3d3cf5bf Oct 30 21:36:17.529461 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:17.529433 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6"] Oct 30 21:36:17.690765 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:17.690721 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6" event={"ID":"28246ca0-71ac-4f01-bc42-52b5afbedc3a","Type":"ContainerStarted","Data":"e248a4c17e6e95552bd2cd40c35dd880bcfb5707f49d7db73996a4174f17ddf3"} Oct 30 21:36:17.693255 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:17.693221 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq" event={"ID":"d6ef610a-9e6f-47cc-9e18-7406909d5df1","Type":"ContainerStarted","Data":"7f7d9453f19bfbd92cbd7b1e3b8eacb0e0bcc6466f867077da49359b3d3cf5bf"} Oct 30 21:36:17.695109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:17.695076 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n" event={"ID":"04df2344-f9e9-4121-887e-0915c6714f8c","Type":"ContainerStarted","Data":"44aa6822f33035b09a249943fb708d9f53330b0684c7f0ad02a2ff2b6e688c72"} Oct 30 21:36:17.741003 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:17.740947 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n" podStartSLOduration=3.740932284 podStartE2EDuration="3.740932284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:17.739343202 +0000 UTC m=+2164.860416949" watchObservedRunningTime="2026-10-30 21:36:17.740932284 +0000 UTC m=+2164.862006029" Oct 30 21:36:18.706388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:18.706349 4926 generic.go:356] "Generic (PLEG): container finished" podID="28246ca0-71ac-4f01-bc42-52b5afbedc3a" containerID="e5d179fac78e48c1dbf86dc4196f7935b664e23e4947b84f5c5fb7669b13e537" exitCode=0 Oct 30 21:36:18.706843 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:18.706435 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6" event={"ID":"28246ca0-71ac-4f01-bc42-52b5afbedc3a","Type":"ContainerDied","Data":"e5d179fac78e48c1dbf86dc4196f7935b664e23e4947b84f5c5fb7669b13e537"} Oct 30 21:36:18.708560 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:18.708534 4926 generic.go:356] "Generic (PLEG): container finished" podID="d6ef610a-9e6f-47cc-9e18-7406909d5df1" containerID="e403b0a0d74482ccd0c2a88833b2db093aeda11dece44a97626c506bb92ebe8a" exitCode=0 Oct 30 21:36:18.708696 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:18.708620 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq" event={"ID":"d6ef610a-9e6f-47cc-9e18-7406909d5df1","Type":"ContainerDied","Data":"e403b0a0d74482ccd0c2a88833b2db093aeda11dece44a97626c506bb92ebe8a"} Oct 30 21:36:18.710407 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:18.710384 4926 generic.go:356] "Generic (PLEG): container finished" podID="04df2344-f9e9-4121-887e-0915c6714f8c" containerID="44aa6822f33035b09a249943fb708d9f53330b0684c7f0ad02a2ff2b6e688c72" exitCode=0 Oct 30 21:36:18.710526 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:18.710414 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n" event={"ID":"04df2344-f9e9-4121-887e-0915c6714f8c","Type":"ContainerDied","Data":"44aa6822f33035b09a249943fb708d9f53330b0684c7f0ad02a2ff2b6e688c72"} Oct 30 21:36:19.952642 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:19.952609 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6" Oct 30 21:36:20.005642 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.005614 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq" Oct 30 21:36:20.017622 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.017574 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28246ca0-71ac-4f01-bc42-52b5afbedc3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28246ca0-71ac-4f01-bc42-52b5afbedc3a-maas-api-openshift-ca\") pod \"28246ca0-71ac-4f01-bc42-52b5afbedc3a\" (UID: \"28246ca0-71ac-4f01-bc42-52b5afbedc3a\") " Oct 30 21:36:20.017803 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.017684 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28246ca0-71ac-4f01-bc42-52b5afbedc3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28246ca0-71ac-4f01-bc42-52b5afbedc3a-maas-api-xks-ca\") pod \"28246ca0-71ac-4f01-bc42-52b5afbedc3a\" (UID: \"28246ca0-71ac-4f01-bc42-52b5afbedc3a\") " Oct 30 21:36:20.017945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.017927 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28246ca0-71ac-4f01-bc42-52b5afbedc3a-kube-api-access-zwc7g\" (UniqueName: \"kubernetes.io/projected/28246ca0-71ac-4f01-bc42-52b5afbedc3a-kube-api-access-zwc7g\") pod \"28246ca0-71ac-4f01-bc42-52b5afbedc3a\" (UID: \"28246ca0-71ac-4f01-bc42-52b5afbedc3a\") " Oct 30 21:36:20.019090 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.019064 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28246ca0-71ac-4f01-bc42-52b5afbedc3a-maas-api-openshift-ca" pod "28246ca0-71ac-4f01-bc42-52b5afbedc3a" (UID: "28246ca0-71ac-4f01-bc42-52b5afbedc3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:20.020630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.020598 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28246ca0-71ac-4f01-bc42-52b5afbedc3a-kube-api-access-zwc7g" pod "28246ca0-71ac-4f01-bc42-52b5afbedc3a" (UID: "28246ca0-71ac-4f01-bc42-52b5afbedc3a"). InnerVolumeSpecName "kube-api-access-zwc7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:20.020850 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.020825 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28246ca0-71ac-4f01-bc42-52b5afbedc3a-maas-api-xks-ca" pod "28246ca0-71ac-4f01-bc42-52b5afbedc3a" (UID: "28246ca0-71ac-4f01-bc42-52b5afbedc3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:36:20.035070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.035024 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n" Oct 30 21:36:20.119857 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.119814 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6ef610a-9e6f-47cc-9e18-7406909d5df1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6ef610a-9e6f-47cc-9e18-7406909d5df1-maas-api-xks-ca\") pod \"d6ef610a-9e6f-47cc-9e18-7406909d5df1\" (UID: \"d6ef610a-9e6f-47cc-9e18-7406909d5df1\") " Oct 30 21:36:20.120082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.119908 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04df2344-f9e9-4121-887e-0915c6714f8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04df2344-f9e9-4121-887e-0915c6714f8c-maas-api-openshift-ca\") pod \"04df2344-f9e9-4121-887e-0915c6714f8c\" (UID: \"04df2344-f9e9-4121-887e-0915c6714f8c\") " Oct 30 21:36:20.120082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.120005 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04df2344-f9e9-4121-887e-0915c6714f8c-kube-api-access-rp8qx\" (UniqueName: \"kubernetes.io/projected/04df2344-f9e9-4121-887e-0915c6714f8c-kube-api-access-rp8qx\") pod \"04df2344-f9e9-4121-887e-0915c6714f8c\" (UID: \"04df2344-f9e9-4121-887e-0915c6714f8c\") " Oct 30 21:36:20.120217 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.120143 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04df2344-f9e9-4121-887e-0915c6714f8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04df2344-f9e9-4121-887e-0915c6714f8c-maas-api-xks-ca\") pod \"04df2344-f9e9-4121-887e-0915c6714f8c\" (UID: \"04df2344-f9e9-4121-887e-0915c6714f8c\") " Oct 30 21:36:20.120282 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.120225 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6ef610a-9e6f-47cc-9e18-7406909d5df1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6ef610a-9e6f-47cc-9e18-7406909d5df1-maas-api-openshift-ca\") pod \"d6ef610a-9e6f-47cc-9e18-7406909d5df1\" (UID: \"d6ef610a-9e6f-47cc-9e18-7406909d5df1\") " Oct 30 21:36:20.120341 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.120320 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6ef610a-9e6f-47cc-9e18-7406909d5df1-kube-api-access-75zst\" (UniqueName: \"kubernetes.io/projected/d6ef610a-9e6f-47cc-9e18-7406909d5df1-kube-api-access-75zst\") pod \"d6ef610a-9e6f-47cc-9e18-7406909d5df1\" (UID: \"d6ef610a-9e6f-47cc-9e18-7406909d5df1\") " Oct 30 21:36:20.121744 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.121416 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04df2344-f9e9-4121-887e-0915c6714f8c-maas-api-openshift-ca" pod "04df2344-f9e9-4121-887e-0915c6714f8c" (UID: "04df2344-f9e9-4121-887e-0915c6714f8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:20.121744 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.121539 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ef610a-9e6f-47cc-9e18-7406909d5df1-maas-api-openshift-ca" pod "d6ef610a-9e6f-47cc-9e18-7406909d5df1" (UID: "d6ef610a-9e6f-47cc-9e18-7406909d5df1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:20.125083 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.124366 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04df2344-f9e9-4121-887e-0915c6714f8c-maas-api-xks-ca" pod "04df2344-f9e9-4121-887e-0915c6714f8c" (UID: "04df2344-f9e9-4121-887e-0915c6714f8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:36:20.125083 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.124871 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6ef610a-9e6f-47cc-9e18-7406909d5df1-maas-api-xks-ca" pod "d6ef610a-9e6f-47cc-9e18-7406909d5df1" (UID: "d6ef610a-9e6f-47cc-9e18-7406909d5df1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:36:20.125479 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.125447 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ef610a-9e6f-47cc-9e18-7406909d5df1-kube-api-access-75zst" pod "d6ef610a-9e6f-47cc-9e18-7406909d5df1" (UID: "d6ef610a-9e6f-47cc-9e18-7406909d5df1"). InnerVolumeSpecName "kube-api-access-75zst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:20.125909 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.125620 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04df2344-f9e9-4121-887e-0915c6714f8c-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:20.125909 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.125669 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6ef610a-9e6f-47cc-9e18-7406909d5df1-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:20.125909 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.125712 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwc7g\" (UniqueName: \"kubernetes.io/projected/28246ca0-71ac-4f01-bc42-52b5afbedc3a-kube-api-access-zwc7g\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:20.125909 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.125728 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28246ca0-71ac-4f01-bc42-52b5afbedc3a-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:20.125909 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.125744 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28246ca0-71ac-4f01-bc42-52b5afbedc3a-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:20.125909 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.125758 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6ef610a-9e6f-47cc-9e18-7406909d5df1-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:20.125909 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.125793 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04df2344-f9e9-4121-887e-0915c6714f8c-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:20.128114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.128080 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04df2344-f9e9-4121-887e-0915c6714f8c-kube-api-access-rp8qx" pod "04df2344-f9e9-4121-887e-0915c6714f8c" (UID: "04df2344-f9e9-4121-887e-0915c6714f8c"). InnerVolumeSpecName "kube-api-access-rp8qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:20.227840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.227804 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp8qx\" (UniqueName: \"kubernetes.io/projected/04df2344-f9e9-4121-887e-0915c6714f8c-kube-api-access-rp8qx\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:20.227840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.227837 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75zst\" (UniqueName: \"kubernetes.io/projected/d6ef610a-9e6f-47cc-9e18-7406909d5df1-kube-api-access-75zst\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:20.730460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.730428 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq" Oct 30 21:36:20.730693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.730435 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq" event={"ID":"d6ef610a-9e6f-47cc-9e18-7406909d5df1","Type":"ContainerDied","Data":"7f7d9453f19bfbd92cbd7b1e3b8eacb0e0bcc6466f867077da49359b3d3cf5bf"} Oct 30 21:36:20.730693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.730583 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f7d9453f19bfbd92cbd7b1e3b8eacb0e0bcc6466f867077da49359b3d3cf5bf" Oct 30 21:36:20.732467 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.732421 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n" Oct 30 21:36:20.732616 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.732564 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n" event={"ID":"04df2344-f9e9-4121-887e-0915c6714f8c","Type":"ContainerDied","Data":"72976ac8df23c6692ece5a3e92104e84629086ae93c04eab75926db86a7911b1"} Oct 30 21:36:20.732616 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.732611 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72976ac8df23c6692ece5a3e92104e84629086ae93c04eab75926db86a7911b1" Oct 30 21:36:20.735430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.735399 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6" event={"ID":"28246ca0-71ac-4f01-bc42-52b5afbedc3a","Type":"ContainerDied","Data":"e248a4c17e6e95552bd2cd40c35dd880bcfb5707f49d7db73996a4174f17ddf3"} Oct 30 21:36:20.735577 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.735450 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e248a4c17e6e95552bd2cd40c35dd880bcfb5707f49d7db73996a4174f17ddf3" Oct 30 21:36:20.735577 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:20.735515 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6" Oct 30 21:36:29.682867 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:29.682826 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh"] Oct 30 21:36:29.691422 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:29.691385 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1eab65-5vdnh"] Oct 30 21:36:31.018987 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.018953 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24d947c8-b5e0-4d7c-8517-c2fd7984424d" path="/var/lib/kubelet/pods/24d947c8-b5e0-4d7c-8517-c2fd7984424d/volumes" Oct 30 21:36:31.403275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.403167 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" podUID="09e24087-be45-4f70-bdc0-5af059800182" containerName="main" containerID="cri-o://de180d1c0b637d03718fcaa34899bdc9578d2aed237ec1bfbcc3b0963e943d2a" gracePeriod=45 Oct 30 21:36:31.472837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.472793 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" podUID="09e24087-be45-4f70-bdc0-5af059800182" containerName="main" probeResult="failure" output="Get \"https://10.132.0.160:8000/ready\": dial tcp 10.132.0.160:8000: connect: connection refused" Oct 30 21:36:31.752648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.752620 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:36:31.777311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.777277 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09e24087-be45-4f70-bdc0-5af059800182-tls-certs\" (UniqueName: \"kubernetes.io/secret/09e24087-be45-4f70-bdc0-5af059800182-tls-certs\") pod \"09e24087-be45-4f70-bdc0-5af059800182\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " Oct 30 21:36:31.777477 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.777457 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09e24087-be45-4f70-bdc0-5af059800182-kube-api-access-55ngd\" (UniqueName: \"kubernetes.io/projected/09e24087-be45-4f70-bdc0-5af059800182-kube-api-access-55ngd\") pod \"09e24087-be45-4f70-bdc0-5af059800182\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " Oct 30 21:36:31.777551 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.777534 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-model-cache\") pod \"09e24087-be45-4f70-bdc0-5af059800182\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " Oct 30 21:36:31.777605 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.777583 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-tmp-dir\") pod \"09e24087-be45-4f70-bdc0-5af059800182\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " Oct 30 21:36:31.777687 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.777633 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-dshm\" (UniqueName: \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-dshm\") pod \"09e24087-be45-4f70-bdc0-5af059800182\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " Oct 30 21:36:31.777750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.777719 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-home\" (UniqueName: \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-home\") pod \"09e24087-be45-4f70-bdc0-5af059800182\" (UID: \"09e24087-be45-4f70-bdc0-5af059800182\") " Oct 30 21:36:31.777961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.777931 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-model-cache" pod "09e24087-be45-4f70-bdc0-5af059800182" (UID: "09e24087-be45-4f70-bdc0-5af059800182"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:31.778111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.778075 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-home" pod "09e24087-be45-4f70-bdc0-5af059800182" (UID: "09e24087-be45-4f70-bdc0-5af059800182"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:31.778193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.778144 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-tmp-dir" pod "09e24087-be45-4f70-bdc0-5af059800182" (UID: "09e24087-be45-4f70-bdc0-5af059800182"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:31.778534 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.778501 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:31.778534 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.778532 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:31.778688 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.778549 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:31.779778 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.779747 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09e24087-be45-4f70-bdc0-5af059800182-tls-certs" pod "09e24087-be45-4f70-bdc0-5af059800182" (UID: "09e24087-be45-4f70-bdc0-5af059800182"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:36:31.779896 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.779879 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-dshm" pod "09e24087-be45-4f70-bdc0-5af059800182" (UID: "09e24087-be45-4f70-bdc0-5af059800182"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:31.779948 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.779889 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e24087-be45-4f70-bdc0-5af059800182-kube-api-access-55ngd" pod "09e24087-be45-4f70-bdc0-5af059800182" (UID: "09e24087-be45-4f70-bdc0-5af059800182"). InnerVolumeSpecName "kube-api-access-55ngd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:31.857149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.848360 4926 generic.go:356] "Generic (PLEG): container finished" podID="09e24087-be45-4f70-bdc0-5af059800182" containerID="de180d1c0b637d03718fcaa34899bdc9578d2aed237ec1bfbcc3b0963e943d2a" exitCode=0 Oct 30 21:36:31.857149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.848407 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" Oct 30 21:36:31.857149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.848428 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" event={"ID":"09e24087-be45-4f70-bdc0-5af059800182","Type":"ContainerDied","Data":"de180d1c0b637d03718fcaa34899bdc9578d2aed237ec1bfbcc3b0963e943d2a"} Oct 30 21:36:31.857149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.848471 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp" event={"ID":"09e24087-be45-4f70-bdc0-5af059800182","Type":"ContainerDied","Data":"4b3d1ce8f461b5a9dc80d13e42af6aff82435d7efa40bdaea30e0725acdbc0a9"} Oct 30 21:36:31.857149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.848494 4926 scope.go:122] "RemoveContainer" containerID="de180d1c0b637d03718fcaa34899bdc9578d2aed237ec1bfbcc3b0963e943d2a" Oct 30 21:36:31.864541 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.864519 4926 scope.go:122] "RemoveContainer" containerID="de180d1c0b637d03718fcaa34899bdc9578d2aed237ec1bfbcc3b0963e943d2a" Oct 30 21:36:31.864883 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:36:31.864864 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de180d1c0b637d03718fcaa34899bdc9578d2aed237ec1bfbcc3b0963e943d2a\": container with ID starting with de180d1c0b637d03718fcaa34899bdc9578d2aed237ec1bfbcc3b0963e943d2a not found: ID does not exist" containerID="de180d1c0b637d03718fcaa34899bdc9578d2aed237ec1bfbcc3b0963e943d2a" Oct 30 21:36:31.864963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.864896 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de180d1c0b637d03718fcaa34899bdc9578d2aed237ec1bfbcc3b0963e943d2a"} err="failed to get container status \"de180d1c0b637d03718fcaa34899bdc9578d2aed237ec1bfbcc3b0963e943d2a\": rpc error: code = NotFound desc = could not find container \"de180d1c0b637d03718fcaa34899bdc9578d2aed237ec1bfbcc3b0963e943d2a\": container with ID starting with de180d1c0b637d03718fcaa34899bdc9578d2aed237ec1bfbcc3b0963e943d2a not found: ID does not exist" Oct 30 21:36:31.880215 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.880171 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09e24087-be45-4f70-bdc0-5af059800182-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:31.880363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.880216 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55ngd\" (UniqueName: \"kubernetes.io/projected/09e24087-be45-4f70-bdc0-5af059800182-kube-api-access-55ngd\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:31.880363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.880238 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09e24087-be45-4f70-bdc0-5af059800182-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:31.908814 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.908770 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp"] Oct 30 21:36:31.921159 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:31.921124 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3b28bc/test-model-3b28bc-kserve-65dbc5c95-vcppp"] Oct 30 21:36:33.022140 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:33.022102 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09e24087-be45-4f70-bdc0-5af059800182" path="/var/lib/kubelet/pods/09e24087-be45-4f70-bdc0-5af059800182/volumes" Oct 30 21:36:35.061134 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:35.061093 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6"] Oct 30 21:36:35.063742 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:35.062058 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" podUID="eff73851-b5a4-4161-9177-5a06b343cec1" containerName="istio-proxy" containerID="cri-o://b26e1d10856de67bb2c4d2cb04f2f9f2e67b73d7b676352d5deda36b82d22a2e" gracePeriod=30 Oct 30 21:36:37.486756 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:36:37.486715 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:36:37.927853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:37.927815 4926 patch_prober.go:29] interesting pod/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:36:37.928075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:37.927880 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" podUID="eff73851-b5a4-4161-9177-5a06b343cec1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:36:38.044477 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.044426 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx"] Oct 30 21:36:38.045048 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.044962 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" podUID="eae78b55-4403-42fc-8646-633c7bbded37" containerName="istio-proxy" containerID="cri-o://fb4e93836eede910d5e9e4b8d70185659de098f4a23a464bc5ca76499659c315" gracePeriod=30 Oct 30 21:36:38.766893 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.766854 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g"] Oct 30 21:36:38.767751 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.767729 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d6ef610a-9e6f-47cc-9e18-7406909d5df1" containerName="revoke-keys" Oct 30 21:36:38.767751 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.767752 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6ef610a-9e6f-47cc-9e18-7406909d5df1" containerName="revoke-keys" Oct 30 21:36:38.767920 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.767806 4926 cpu_manager.go:414] "containerMap: removing container" podUID="04df2344-f9e9-4121-887e-0915c6714f8c" containerName="revoke-keys" Oct 30 21:36:38.767920 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.767814 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04df2344-f9e9-4121-887e-0915c6714f8c" containerName="revoke-keys" Oct 30 21:36:38.767920 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.767844 4926 cpu_manager.go:414] "containerMap: removing container" podUID="28246ca0-71ac-4f01-bc42-52b5afbedc3a" containerName="revoke-keys" Oct 30 21:36:38.767920 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.767852 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28246ca0-71ac-4f01-bc42-52b5afbedc3a" containerName="revoke-keys" Oct 30 21:36:38.767920 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.767862 4926 cpu_manager.go:414] "containerMap: removing container" podUID="09e24087-be45-4f70-bdc0-5af059800182" containerName="main" Oct 30 21:36:38.767920 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.767869 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09e24087-be45-4f70-bdc0-5af059800182" containerName="main" Oct 30 21:36:38.768285 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.768073 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="04df2344-f9e9-4121-887e-0915c6714f8c" containerName="revoke-keys" Oct 30 21:36:38.768285 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.768094 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="28246ca0-71ac-4f01-bc42-52b5afbedc3a" containerName="revoke-keys" Oct 30 21:36:38.768285 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.768107 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="09e24087-be45-4f70-bdc0-5af059800182" containerName="main" Oct 30 21:36:38.768285 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.768123 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6ef610a-9e6f-47cc-9e18-7406909d5df1" containerName="revoke-keys" Oct 30 21:36:38.772181 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.772157 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g" Oct 30 21:36:38.792085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.792047 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g"] Oct 30 21:36:38.798528 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.798497 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:36:38.798847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.798826 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:36:38.895664 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.895626 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9hfz\" (UniqueName: \"kubernetes.io/projected/dd0feb74-1a47-45c7-8493-c420a98692a9-kube-api-access-z9hfz\") pod \"maas-api-revoke-keys-e2e-inf-b-290ede-4q96g\" (UID: \"dd0feb74-1a47-45c7-8493-c420a98692a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g" Oct 30 21:36:38.895893 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.895716 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0feb74-1a47-45c7-8493-c420a98692a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-290ede-4q96g\" (UID: \"dd0feb74-1a47-45c7-8493-c420a98692a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g" Oct 30 21:36:38.895893 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.895860 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0feb74-1a47-45c7-8493-c420a98692a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-290ede-4q96g\" (UID: \"dd0feb74-1a47-45c7-8493-c420a98692a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g" Oct 30 21:36:38.996979 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.996939 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9hfz\" (UniqueName: \"kubernetes.io/projected/dd0feb74-1a47-45c7-8493-c420a98692a9-kube-api-access-z9hfz\") pod \"maas-api-revoke-keys-e2e-inf-b-290ede-4q96g\" (UID: \"dd0feb74-1a47-45c7-8493-c420a98692a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g" Oct 30 21:36:38.997250 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.997071 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0feb74-1a47-45c7-8493-c420a98692a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-290ede-4q96g\" (UID: \"dd0feb74-1a47-45c7-8493-c420a98692a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g" Oct 30 21:36:38.997250 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.997202 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0feb74-1a47-45c7-8493-c420a98692a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-290ede-4q96g\" (UID: \"dd0feb74-1a47-45c7-8493-c420a98692a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g" Oct 30 21:36:38.997728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.997706 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0feb74-1a47-45c7-8493-c420a98692a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-290ede-4q96g\" (UID: \"dd0feb74-1a47-45c7-8493-c420a98692a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g" Oct 30 21:36:38.999630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:38.999605 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0feb74-1a47-45c7-8493-c420a98692a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-290ede-4q96g\" (UID: \"dd0feb74-1a47-45c7-8493-c420a98692a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g" Oct 30 21:36:39.020616 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:39.020545 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9hfz\" (UniqueName: \"kubernetes.io/projected/dd0feb74-1a47-45c7-8493-c420a98692a9-kube-api-access-z9hfz\") pod \"maas-api-revoke-keys-e2e-inf-b-290ede-4q96g\" (UID: \"dd0feb74-1a47-45c7-8493-c420a98692a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g" Oct 30 21:36:39.094087 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:39.089808 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g" Oct 30 21:36:39.419975 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:39.419941 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g"] Oct 30 21:36:39.420549 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:36:39.420519 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0feb74_1a47_45c7_8493_c420a98692a9.slice/crio-d93072d1846ce36634c36b1f2eb260ea4e8166e46b4ab0b5c983e8f468fdd3e7 WatchSource:0}: Error finding container d93072d1846ce36634c36b1f2eb260ea4e8166e46b4ab0b5c983e8f468fdd3e7: Status 404 returned error can't find the container with id d93072d1846ce36634c36b1f2eb260ea4e8166e46b4ab0b5c983e8f468fdd3e7 Oct 30 21:36:39.935291 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:39.935253 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g" event={"ID":"dd0feb74-1a47-45c7-8493-c420a98692a9","Type":"ContainerStarted","Data":"36676f3b63b4a670fa3045cc28889f12dfae32eba71e756b9042a6bfee4b9dfc"} Oct 30 21:36:39.935717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:39.935320 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g" event={"ID":"dd0feb74-1a47-45c7-8493-c420a98692a9","Type":"ContainerStarted","Data":"d93072d1846ce36634c36b1f2eb260ea4e8166e46b4ab0b5c983e8f468fdd3e7"} Oct 30 21:36:39.994144 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:39.993217 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g" podStartSLOduration=1.993199232 podStartE2EDuration="1.993199232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:39.991764896 +0000 UTC m=+2187.112838641" watchObservedRunningTime="2026-10-30 21:36:39.993199232 +0000 UTC m=+2187.114272980" Oct 30 21:36:40.482293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.482262 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:36:40.523891 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.523857 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eff73851-b5a4-4161-9177-5a06b343cec1-istio-token\" (UniqueName: \"kubernetes.io/projected/eff73851-b5a4-4161-9177-5a06b343cec1-istio-token\") pod \"eff73851-b5a4-4161-9177-5a06b343cec1\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " Oct 30 21:36:40.524105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.523918 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eff73851-b5a4-4161-9177-5a06b343cec1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eff73851-b5a4-4161-9177-5a06b343cec1-istio-podinfo\") pod \"eff73851-b5a4-4161-9177-5a06b343cec1\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " Oct 30 21:36:40.524105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.523965 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-istio-data\") pod \"eff73851-b5a4-4161-9177-5a06b343cec1\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " Oct 30 21:36:40.524105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.524023 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eff73851-b5a4-4161-9177-5a06b343cec1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eff73851-b5a4-4161-9177-5a06b343cec1-istiod-ca-cert\") pod \"eff73851-b5a4-4161-9177-5a06b343cec1\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " Oct 30 21:36:40.524234 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.524182 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-istio-envoy\") pod \"eff73851-b5a4-4161-9177-5a06b343cec1\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " Oct 30 21:36:40.524358 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.524330 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-workload-certs\") pod \"eff73851-b5a4-4161-9177-5a06b343cec1\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " Oct 30 21:36:40.524446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.524392 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-workload-socket\") pod \"eff73851-b5a4-4161-9177-5a06b343cec1\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " Oct 30 21:36:40.524502 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.524457 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eff73851-b5a4-4161-9177-5a06b343cec1-kube-api-access-6vddk\" (UniqueName: \"kubernetes.io/projected/eff73851-b5a4-4161-9177-5a06b343cec1-kube-api-access-6vddk\") pod \"eff73851-b5a4-4161-9177-5a06b343cec1\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " Oct 30 21:36:40.524573 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.524552 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-credential-socket\") pod \"eff73851-b5a4-4161-9177-5a06b343cec1\" (UID: \"eff73851-b5a4-4161-9177-5a06b343cec1\") " Oct 30 21:36:40.524722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.524595 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-workload-certs" pod "eff73851-b5a4-4161-9177-5a06b343cec1" (UID: "eff73851-b5a4-4161-9177-5a06b343cec1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:40.524722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.524606 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-istio-data" pod "eff73851-b5a4-4161-9177-5a06b343cec1" (UID: "eff73851-b5a4-4161-9177-5a06b343cec1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:40.524722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.524629 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff73851-b5a4-4161-9177-5a06b343cec1-istiod-ca-cert" pod "eff73851-b5a4-4161-9177-5a06b343cec1" (UID: "eff73851-b5a4-4161-9177-5a06b343cec1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:40.524889 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.524791 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-workload-socket" pod "eff73851-b5a4-4161-9177-5a06b343cec1" (UID: "eff73851-b5a4-4161-9177-5a06b343cec1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:40.524944 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.524901 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-credential-socket" pod "eff73851-b5a4-4161-9177-5a06b343cec1" (UID: "eff73851-b5a4-4161-9177-5a06b343cec1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:40.525394 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.525367 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:40.525485 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.525398 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eff73851-b5a4-4161-9177-5a06b343cec1-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:40.525485 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.525414 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:40.525485 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.525429 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:40.525485 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.525441 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:40.526987 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.526960 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff73851-b5a4-4161-9177-5a06b343cec1-istio-token" pod "eff73851-b5a4-4161-9177-5a06b343cec1" (UID: "eff73851-b5a4-4161-9177-5a06b343cec1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:40.527124 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.527107 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff73851-b5a4-4161-9177-5a06b343cec1-kube-api-access-6vddk" pod "eff73851-b5a4-4161-9177-5a06b343cec1" (UID: "eff73851-b5a4-4161-9177-5a06b343cec1"). InnerVolumeSpecName "kube-api-access-6vddk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:40.527275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.527255 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eff73851-b5a4-4161-9177-5a06b343cec1-istio-podinfo" pod "eff73851-b5a4-4161-9177-5a06b343cec1" (UID: "eff73851-b5a4-4161-9177-5a06b343cec1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:36:40.527764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.527744 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-istio-envoy" pod "eff73851-b5a4-4161-9177-5a06b343cec1" (UID: "eff73851-b5a4-4161-9177-5a06b343cec1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:40.627461 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.627423 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eff73851-b5a4-4161-9177-5a06b343cec1-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:40.627461 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.627456 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eff73851-b5a4-4161-9177-5a06b343cec1-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:40.627461 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.627465 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vddk\" (UniqueName: \"kubernetes.io/projected/eff73851-b5a4-4161-9177-5a06b343cec1-kube-api-access-6vddk\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:40.627733 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.627475 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eff73851-b5a4-4161-9177-5a06b343cec1-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:40.948311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.948271 4926 generic.go:356] "Generic (PLEG): container finished" podID="eff73851-b5a4-4161-9177-5a06b343cec1" containerID="b26e1d10856de67bb2c4d2cb04f2f9f2e67b73d7b676352d5deda36b82d22a2e" exitCode=0 Oct 30 21:36:40.948869 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.948326 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" Oct 30 21:36:40.948869 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.948352 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" event={"ID":"eff73851-b5a4-4161-9177-5a06b343cec1","Type":"ContainerDied","Data":"b26e1d10856de67bb2c4d2cb04f2f9f2e67b73d7b676352d5deda36b82d22a2e"} Oct 30 21:36:40.948869 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.948406 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6" event={"ID":"eff73851-b5a4-4161-9177-5a06b343cec1","Type":"ContainerDied","Data":"07dea4ca715e9ca39e463e1a3b43dc1b608f9ada72b9ca52c80ded9f9f6b4eb4"} Oct 30 21:36:40.948869 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.948435 4926 scope.go:122] "RemoveContainer" containerID="b26e1d10856de67bb2c4d2cb04f2f9f2e67b73d7b676352d5deda36b82d22a2e" Oct 30 21:36:40.950414 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.950391 4926 generic.go:356] "Generic (PLEG): container finished" podID="dd0feb74-1a47-45c7-8493-c420a98692a9" containerID="36676f3b63b4a670fa3045cc28889f12dfae32eba71e756b9042a6bfee4b9dfc" exitCode=0 Oct 30 21:36:40.950553 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.950465 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g" event={"ID":"dd0feb74-1a47-45c7-8493-c420a98692a9","Type":"ContainerDied","Data":"36676f3b63b4a670fa3045cc28889f12dfae32eba71e756b9042a6bfee4b9dfc"} Oct 30 21:36:40.960239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.960219 4926 scope.go:122] "RemoveContainer" containerID="b26e1d10856de67bb2c4d2cb04f2f9f2e67b73d7b676352d5deda36b82d22a2e" Oct 30 21:36:40.960510 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:36:40.960491 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b26e1d10856de67bb2c4d2cb04f2f9f2e67b73d7b676352d5deda36b82d22a2e\": container with ID starting with b26e1d10856de67bb2c4d2cb04f2f9f2e67b73d7b676352d5deda36b82d22a2e not found: ID does not exist" containerID="b26e1d10856de67bb2c4d2cb04f2f9f2e67b73d7b676352d5deda36b82d22a2e" Oct 30 21:36:40.960559 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:40.960518 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26e1d10856de67bb2c4d2cb04f2f9f2e67b73d7b676352d5deda36b82d22a2e"} err="failed to get container status \"b26e1d10856de67bb2c4d2cb04f2f9f2e67b73d7b676352d5deda36b82d22a2e\": rpc error: code = NotFound desc = could not find container \"b26e1d10856de67bb2c4d2cb04f2f9f2e67b73d7b676352d5deda36b82d22a2e\": container with ID starting with b26e1d10856de67bb2c4d2cb04f2f9f2e67b73d7b676352d5deda36b82d22a2e not found: ID does not exist" Oct 30 21:36:41.000108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:41.000065 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6"] Oct 30 21:36:41.019256 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:41.019225 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-56ac74-openshift-default-859dc8d675-wzvb6"] Oct 30 21:36:41.535215 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:41.535176 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p"] Oct 30 21:36:41.535844 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:41.535830 4926 cpu_manager.go:414] "containerMap: removing container" podUID="eff73851-b5a4-4161-9177-5a06b343cec1" containerName="istio-proxy" Oct 30 21:36:41.535891 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:41.535845 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eff73851-b5a4-4161-9177-5a06b343cec1" containerName="istio-proxy" Oct 30 21:36:41.535959 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:41.535949 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="eff73851-b5a4-4161-9177-5a06b343cec1" containerName="istio-proxy" Oct 30 21:36:41.539478 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:41.539456 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p" Oct 30 21:36:41.573955 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:41.573919 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p"] Oct 30 21:36:41.641085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:41.641049 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw65b\" (UniqueName: \"kubernetes.io/projected/bbccc867-07a9-4f40-a181-bd0a6e42571e-kube-api-access-mw65b\") pod \"maas-api-revoke-sub-4b8dbb49-7r99p\" (UID: \"bbccc867-07a9-4f40-a181-bd0a6e42571e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p" Oct 30 21:36:41.641309 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:41.641162 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbccc867-07a9-4f40-a181-bd0a6e42571e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b8dbb49-7r99p\" (UID: \"bbccc867-07a9-4f40-a181-bd0a6e42571e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p" Oct 30 21:36:41.641309 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:41.641219 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbccc867-07a9-4f40-a181-bd0a6e42571e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b8dbb49-7r99p\" (UID: \"bbccc867-07a9-4f40-a181-bd0a6e42571e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p" Oct 30 21:36:41.742596 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:41.742558 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw65b\" (UniqueName: \"kubernetes.io/projected/bbccc867-07a9-4f40-a181-bd0a6e42571e-kube-api-access-mw65b\") pod \"maas-api-revoke-sub-4b8dbb49-7r99p\" (UID: \"bbccc867-07a9-4f40-a181-bd0a6e42571e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p" Oct 30 21:36:41.742817 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:41.742607 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbccc867-07a9-4f40-a181-bd0a6e42571e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b8dbb49-7r99p\" (UID: \"bbccc867-07a9-4f40-a181-bd0a6e42571e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p" Oct 30 21:36:41.742817 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:41.742636 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbccc867-07a9-4f40-a181-bd0a6e42571e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b8dbb49-7r99p\" (UID: \"bbccc867-07a9-4f40-a181-bd0a6e42571e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p" Oct 30 21:36:41.743401 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:41.743361 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbccc867-07a9-4f40-a181-bd0a6e42571e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b8dbb49-7r99p\" (UID: \"bbccc867-07a9-4f40-a181-bd0a6e42571e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p" Oct 30 21:36:41.745217 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:41.745189 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbccc867-07a9-4f40-a181-bd0a6e42571e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b8dbb49-7r99p\" (UID: \"bbccc867-07a9-4f40-a181-bd0a6e42571e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p" Oct 30 21:36:41.762582 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:41.762556 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw65b\" (UniqueName: \"kubernetes.io/projected/bbccc867-07a9-4f40-a181-bd0a6e42571e-kube-api-access-mw65b\") pod \"maas-api-revoke-sub-4b8dbb49-7r99p\" (UID: \"bbccc867-07a9-4f40-a181-bd0a6e42571e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p" Oct 30 21:36:41.855177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:41.855091 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p" Oct 30 21:36:42.162830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:42.162801 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g" Oct 30 21:36:42.177467 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:42.177436 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p"] Oct 30 21:36:42.179525 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:36:42.179488 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbccc867_07a9_4f40_a181_bd0a6e42571e.slice/crio-06d077defa0f8ed3e6ee072c4dcfa49067362636600c8045517f7c77e75d5565 WatchSource:0}: Error finding container 06d077defa0f8ed3e6ee072c4dcfa49067362636600c8045517f7c77e75d5565: Status 404 returned error can't find the container with id 06d077defa0f8ed3e6ee072c4dcfa49067362636600c8045517f7c77e75d5565 Oct 30 21:36:42.252502 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:42.252474 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd0feb74-1a47-45c7-8493-c420a98692a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0feb74-1a47-45c7-8493-c420a98692a9-maas-api-openshift-ca\") pod \"dd0feb74-1a47-45c7-8493-c420a98692a9\" (UID: \"dd0feb74-1a47-45c7-8493-c420a98692a9\") " Oct 30 21:36:42.252682 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:42.252587 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd0feb74-1a47-45c7-8493-c420a98692a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0feb74-1a47-45c7-8493-c420a98692a9-maas-api-xks-ca\") pod \"dd0feb74-1a47-45c7-8493-c420a98692a9\" (UID: \"dd0feb74-1a47-45c7-8493-c420a98692a9\") " Oct 30 21:36:42.252766 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:42.252733 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0feb74-1a47-45c7-8493-c420a98692a9-kube-api-access-z9hfz\" (UniqueName: \"kubernetes.io/projected/dd0feb74-1a47-45c7-8493-c420a98692a9-kube-api-access-z9hfz\") pod \"dd0feb74-1a47-45c7-8493-c420a98692a9\" (UID: \"dd0feb74-1a47-45c7-8493-c420a98692a9\") " Oct 30 21:36:42.252933 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:42.252905 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0feb74-1a47-45c7-8493-c420a98692a9-maas-api-openshift-ca" pod "dd0feb74-1a47-45c7-8493-c420a98692a9" (UID: "dd0feb74-1a47-45c7-8493-c420a98692a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:42.253368 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:42.253344 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd0feb74-1a47-45c7-8493-c420a98692a9-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:42.254761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:42.254737 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd0feb74-1a47-45c7-8493-c420a98692a9-maas-api-xks-ca" pod "dd0feb74-1a47-45c7-8493-c420a98692a9" (UID: "dd0feb74-1a47-45c7-8493-c420a98692a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:36:42.254827 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:42.254793 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0feb74-1a47-45c7-8493-c420a98692a9-kube-api-access-z9hfz" pod "dd0feb74-1a47-45c7-8493-c420a98692a9" (UID: "dd0feb74-1a47-45c7-8493-c420a98692a9"). InnerVolumeSpecName "kube-api-access-z9hfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:42.355275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:42.355236 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd0feb74-1a47-45c7-8493-c420a98692a9-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:42.355275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:42.355270 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9hfz\" (UniqueName: \"kubernetes.io/projected/dd0feb74-1a47-45c7-8493-c420a98692a9-kube-api-access-z9hfz\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:42.973077 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:42.973001 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p" event={"ID":"bbccc867-07a9-4f40-a181-bd0a6e42571e","Type":"ContainerStarted","Data":"4d1eeeac674ddaca267e80cbdf1e231d03a2a86dc9d2d45a85730b7e027ce3ec"} Oct 30 21:36:42.973306 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:42.973087 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p" event={"ID":"bbccc867-07a9-4f40-a181-bd0a6e42571e","Type":"ContainerStarted","Data":"06d077defa0f8ed3e6ee072c4dcfa49067362636600c8045517f7c77e75d5565"} Oct 30 21:36:42.974977 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:42.974948 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g" Oct 30 21:36:42.975151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:42.974977 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g" event={"ID":"dd0feb74-1a47-45c7-8493-c420a98692a9","Type":"ContainerDied","Data":"d93072d1846ce36634c36b1f2eb260ea4e8166e46b4ab0b5c983e8f468fdd3e7"} Oct 30 21:36:42.975151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:42.975069 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d93072d1846ce36634c36b1f2eb260ea4e8166e46b4ab0b5c983e8f468fdd3e7" Oct 30 21:36:43.019998 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.019965 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eff73851-b5a4-4161-9177-5a06b343cec1" path="/var/lib/kubelet/pods/eff73851-b5a4-4161-9177-5a06b343cec1/volumes" Oct 30 21:36:43.031241 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.031182 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p" podStartSLOduration=2.031164082 podStartE2EDuration="2.031164082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:43.025256794 +0000 UTC m=+2190.146330539" watchObservedRunningTime="2026-10-30 21:36:43.031164082 +0000 UTC m=+2190.152237828" Oct 30 21:36:43.348752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.348723 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:36:43.469974 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.469925 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-workload-certs\") pod \"eae78b55-4403-42fc-8646-633c7bbded37\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " Oct 30 21:36:43.470217 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.470001 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eae78b55-4403-42fc-8646-633c7bbded37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae78b55-4403-42fc-8646-633c7bbded37-istiod-ca-cert\") pod \"eae78b55-4403-42fc-8646-633c7bbded37\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " Oct 30 21:36:43.470217 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.470109 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae78b55-4403-42fc-8646-633c7bbded37-istio-token\" (UniqueName: \"kubernetes.io/projected/eae78b55-4403-42fc-8646-633c7bbded37-istio-token\") pod \"eae78b55-4403-42fc-8646-633c7bbded37\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " Oct 30 21:36:43.470336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.470238 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-istio-envoy\") pod \"eae78b55-4403-42fc-8646-633c7bbded37\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " Oct 30 21:36:43.470336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.470274 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-workload-certs" pod "eae78b55-4403-42fc-8646-633c7bbded37" (UID: "eae78b55-4403-42fc-8646-633c7bbded37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:43.470336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.470299 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eae78b55-4403-42fc-8646-633c7bbded37-kube-api-access-4h4tw\" (UniqueName: \"kubernetes.io/projected/eae78b55-4403-42fc-8646-633c7bbded37-kube-api-access-4h4tw\") pod \"eae78b55-4403-42fc-8646-633c7bbded37\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " Oct 30 21:36:43.470484 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.470446 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-credential-socket\") pod \"eae78b55-4403-42fc-8646-633c7bbded37\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " Oct 30 21:36:43.470564 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.470546 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-istio-data\") pod \"eae78b55-4403-42fc-8646-633c7bbded37\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " Oct 30 21:36:43.470620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.470599 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-workload-socket\") pod \"eae78b55-4403-42fc-8646-633c7bbded37\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " Oct 30 21:36:43.470669 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.470639 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eae78b55-4403-42fc-8646-633c7bbded37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae78b55-4403-42fc-8646-633c7bbded37-istio-podinfo\") pod \"eae78b55-4403-42fc-8646-633c7bbded37\" (UID: \"eae78b55-4403-42fc-8646-633c7bbded37\") " Oct 30 21:36:43.470669 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.470647 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eae78b55-4403-42fc-8646-633c7bbded37-istiod-ca-cert" pod "eae78b55-4403-42fc-8646-633c7bbded37" (UID: "eae78b55-4403-42fc-8646-633c7bbded37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:43.470888 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.470861 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-workload-socket" pod "eae78b55-4403-42fc-8646-633c7bbded37" (UID: "eae78b55-4403-42fc-8646-633c7bbded37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:43.470941 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.470908 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-istio-data" pod "eae78b55-4403-42fc-8646-633c7bbded37" (UID: "eae78b55-4403-42fc-8646-633c7bbded37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:43.471342 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.471153 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-credential-socket" pod "eae78b55-4403-42fc-8646-633c7bbded37" (UID: "eae78b55-4403-42fc-8646-633c7bbded37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:43.471471 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.471354 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:43.471471 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.471375 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:43.471471 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.471389 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:43.471471 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.471400 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:43.471471 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.471411 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eae78b55-4403-42fc-8646-633c7bbded37-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:43.473652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.473626 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae78b55-4403-42fc-8646-633c7bbded37-istio-token" pod "eae78b55-4403-42fc-8646-633c7bbded37" (UID: "eae78b55-4403-42fc-8646-633c7bbded37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:43.475598 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.475545 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eae78b55-4403-42fc-8646-633c7bbded37-istio-podinfo" pod "eae78b55-4403-42fc-8646-633c7bbded37" (UID: "eae78b55-4403-42fc-8646-633c7bbded37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:36:43.476129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.476100 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-istio-envoy" pod "eae78b55-4403-42fc-8646-633c7bbded37" (UID: "eae78b55-4403-42fc-8646-633c7bbded37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:36:43.476306 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.476281 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eae78b55-4403-42fc-8646-633c7bbded37-kube-api-access-4h4tw" pod "eae78b55-4403-42fc-8646-633c7bbded37" (UID: "eae78b55-4403-42fc-8646-633c7bbded37"). InnerVolumeSpecName "kube-api-access-4h4tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:43.572731 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.572692 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eae78b55-4403-42fc-8646-633c7bbded37-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:43.572731 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.572731 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h4tw\" (UniqueName: \"kubernetes.io/projected/eae78b55-4403-42fc-8646-633c7bbded37-kube-api-access-4h4tw\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:43.572960 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.572748 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eae78b55-4403-42fc-8646-633c7bbded37-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:43.572960 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.572776 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eae78b55-4403-42fc-8646-633c7bbded37-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:43.986107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.986063 4926 generic.go:356] "Generic (PLEG): container finished" podID="bbccc867-07a9-4f40-a181-bd0a6e42571e" containerID="4d1eeeac674ddaca267e80cbdf1e231d03a2a86dc9d2d45a85730b7e027ce3ec" exitCode=0 Oct 30 21:36:43.986433 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.986172 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p" event={"ID":"bbccc867-07a9-4f40-a181-bd0a6e42571e","Type":"ContainerDied","Data":"4d1eeeac674ddaca267e80cbdf1e231d03a2a86dc9d2d45a85730b7e027ce3ec"} Oct 30 21:36:43.987955 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.987924 4926 generic.go:356] "Generic (PLEG): container finished" podID="eae78b55-4403-42fc-8646-633c7bbded37" containerID="fb4e93836eede910d5e9e4b8d70185659de098f4a23a464bc5ca76499659c315" exitCode=0 Oct 30 21:36:43.988149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.987965 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" Oct 30 21:36:43.988149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.988007 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" event={"ID":"eae78b55-4403-42fc-8646-633c7bbded37","Type":"ContainerDied","Data":"fb4e93836eede910d5e9e4b8d70185659de098f4a23a464bc5ca76499659c315"} Oct 30 21:36:43.988149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.988091 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx" event={"ID":"eae78b55-4403-42fc-8646-633c7bbded37","Type":"ContainerDied","Data":"04fafa254197f6d25876d2c5227689a6b6320b6dee9a513b03113e70a123105d"} Oct 30 21:36:43.988149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:43.988122 4926 scope.go:122] "RemoveContainer" containerID="fb4e93836eede910d5e9e4b8d70185659de098f4a23a464bc5ca76499659c315" Oct 30 21:36:44.002556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:44.002536 4926 scope.go:122] "RemoveContainer" containerID="fb4e93836eede910d5e9e4b8d70185659de098f4a23a464bc5ca76499659c315" Oct 30 21:36:44.002848 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:36:44.002820 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4e93836eede910d5e9e4b8d70185659de098f4a23a464bc5ca76499659c315\": container with ID starting with fb4e93836eede910d5e9e4b8d70185659de098f4a23a464bc5ca76499659c315 not found: ID does not exist" containerID="fb4e93836eede910d5e9e4b8d70185659de098f4a23a464bc5ca76499659c315" Oct 30 21:36:44.002929 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:44.002861 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4e93836eede910d5e9e4b8d70185659de098f4a23a464bc5ca76499659c315"} err="failed to get container status \"fb4e93836eede910d5e9e4b8d70185659de098f4a23a464bc5ca76499659c315\": rpc error: code = NotFound desc = could not find container \"fb4e93836eede910d5e9e4b8d70185659de098f4a23a464bc5ca76499659c315\": container with ID starting with fb4e93836eede910d5e9e4b8d70185659de098f4a23a464bc5ca76499659c315 not found: ID does not exist" Oct 30 21:36:44.087984 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:44.087946 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx"] Oct 30 21:36:44.099313 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:44.099276 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1eab65-openshift-default-7c57f54487-qq6vx"] Oct 30 21:36:45.030583 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:45.030543 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eae78b55-4403-42fc-8646-633c7bbded37" path="/var/lib/kubelet/pods/eae78b55-4403-42fc-8646-633c7bbded37/volumes" Oct 30 21:36:45.200173 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:45.200144 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p" Oct 30 21:36:45.309430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:45.309332 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbccc867-07a9-4f40-a181-bd0a6e42571e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbccc867-07a9-4f40-a181-bd0a6e42571e-maas-api-xks-ca\") pod \"bbccc867-07a9-4f40-a181-bd0a6e42571e\" (UID: \"bbccc867-07a9-4f40-a181-bd0a6e42571e\") " Oct 30 21:36:45.309614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:45.309481 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbccc867-07a9-4f40-a181-bd0a6e42571e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbccc867-07a9-4f40-a181-bd0a6e42571e-maas-api-openshift-ca\") pod \"bbccc867-07a9-4f40-a181-bd0a6e42571e\" (UID: \"bbccc867-07a9-4f40-a181-bd0a6e42571e\") " Oct 30 21:36:45.309614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:45.309599 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbccc867-07a9-4f40-a181-bd0a6e42571e-kube-api-access-mw65b\" (UniqueName: \"kubernetes.io/projected/bbccc867-07a9-4f40-a181-bd0a6e42571e-kube-api-access-mw65b\") pod \"bbccc867-07a9-4f40-a181-bd0a6e42571e\" (UID: \"bbccc867-07a9-4f40-a181-bd0a6e42571e\") " Oct 30 21:36:45.309887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:45.309858 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbccc867-07a9-4f40-a181-bd0a6e42571e-maas-api-openshift-ca" pod "bbccc867-07a9-4f40-a181-bd0a6e42571e" (UID: "bbccc867-07a9-4f40-a181-bd0a6e42571e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:45.310339 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:45.310312 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbccc867-07a9-4f40-a181-bd0a6e42571e-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:45.311677 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:45.311643 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbccc867-07a9-4f40-a181-bd0a6e42571e-maas-api-xks-ca" pod "bbccc867-07a9-4f40-a181-bd0a6e42571e" (UID: "bbccc867-07a9-4f40-a181-bd0a6e42571e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:36:45.311806 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:45.311786 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbccc867-07a9-4f40-a181-bd0a6e42571e-kube-api-access-mw65b" pod "bbccc867-07a9-4f40-a181-bd0a6e42571e" (UID: "bbccc867-07a9-4f40-a181-bd0a6e42571e"). InnerVolumeSpecName "kube-api-access-mw65b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:45.412121 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:45.412080 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw65b\" (UniqueName: \"kubernetes.io/projected/bbccc867-07a9-4f40-a181-bd0a6e42571e-kube-api-access-mw65b\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:45.412121 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:45.412116 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbccc867-07a9-4f40-a181-bd0a6e42571e-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:46.009092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:46.009024 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p" Oct 30 21:36:46.009301 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:46.009063 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p" event={"ID":"bbccc867-07a9-4f40-a181-bd0a6e42571e","Type":"ContainerDied","Data":"06d077defa0f8ed3e6ee072c4dcfa49067362636600c8045517f7c77e75d5565"} Oct 30 21:36:46.009373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:46.009301 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06d077defa0f8ed3e6ee072c4dcfa49067362636600c8045517f7c77e75d5565" Oct 30 21:36:51.306245 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:51.302225 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s"] Oct 30 21:36:52.152116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.152076 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa1493c0-4vgv6"] Oct 30 21:36:52.152854 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.152829 4926 cpu_manager.go:414] "containerMap: removing container" podUID="eae78b55-4403-42fc-8646-633c7bbded37" containerName="istio-proxy" Oct 30 21:36:52.152854 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.152852 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eae78b55-4403-42fc-8646-633c7bbded37" containerName="istio-proxy" Oct 30 21:36:52.153064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.152875 4926 cpu_manager.go:414] "containerMap: removing container" podUID="bbccc867-07a9-4f40-a181-bd0a6e42571e" containerName="revoke-keys" Oct 30 21:36:52.153064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.152883 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbccc867-07a9-4f40-a181-bd0a6e42571e" containerName="revoke-keys" Oct 30 21:36:52.153064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.152898 4926 cpu_manager.go:414] "containerMap: removing container" podUID="dd0feb74-1a47-45c7-8493-c420a98692a9" containerName="revoke-keys" Oct 30 21:36:52.153064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.152906 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0feb74-1a47-45c7-8493-c420a98692a9" containerName="revoke-keys" Oct 30 21:36:52.153274 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.153127 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="eae78b55-4403-42fc-8646-633c7bbded37" containerName="istio-proxy" Oct 30 21:36:52.153274 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.153147 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbccc867-07a9-4f40-a181-bd0a6e42571e" containerName="revoke-keys" Oct 30 21:36:52.153274 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.153159 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0feb74-1a47-45c7-8493-c420a98692a9" containerName="revoke-keys" Oct 30 21:36:52.156823 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.156793 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1493c0-4vgv6" Oct 30 21:36:52.195361 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.195332 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:36:52.195721 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.195701 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:36:52.202087 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.202052 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa1493c0-4vgv6"] Oct 30 21:36:52.331928 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.331892 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pwm6\" (UniqueName: \"kubernetes.io/projected/482c7875-59c8-4eed-a756-016214ea4127-kube-api-access-5pwm6\") pod \"maas-api-revoke-sub-fa1493c0-4vgv6\" (UID: \"482c7875-59c8-4eed-a756-016214ea4127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1493c0-4vgv6" Oct 30 21:36:52.332786 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.332047 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482c7875-59c8-4eed-a756-016214ea4127-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa1493c0-4vgv6\" (UID: \"482c7875-59c8-4eed-a756-016214ea4127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1493c0-4vgv6" Oct 30 21:36:52.332786 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.332141 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482c7875-59c8-4eed-a756-016214ea4127-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa1493c0-4vgv6\" (UID: \"482c7875-59c8-4eed-a756-016214ea4127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1493c0-4vgv6" Oct 30 21:36:52.439123 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.435049 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482c7875-59c8-4eed-a756-016214ea4127-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa1493c0-4vgv6\" (UID: \"482c7875-59c8-4eed-a756-016214ea4127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1493c0-4vgv6" Oct 30 21:36:52.439123 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.435165 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482c7875-59c8-4eed-a756-016214ea4127-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa1493c0-4vgv6\" (UID: \"482c7875-59c8-4eed-a756-016214ea4127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1493c0-4vgv6" Oct 30 21:36:52.439123 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.435378 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pwm6\" (UniqueName: \"kubernetes.io/projected/482c7875-59c8-4eed-a756-016214ea4127-kube-api-access-5pwm6\") pod \"maas-api-revoke-sub-fa1493c0-4vgv6\" (UID: \"482c7875-59c8-4eed-a756-016214ea4127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1493c0-4vgv6" Oct 30 21:36:52.439123 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.436553 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482c7875-59c8-4eed-a756-016214ea4127-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa1493c0-4vgv6\" (UID: \"482c7875-59c8-4eed-a756-016214ea4127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1493c0-4vgv6" Oct 30 21:36:52.463760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.463729 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pwm6\" (UniqueName: \"kubernetes.io/projected/482c7875-59c8-4eed-a756-016214ea4127-kube-api-access-5pwm6\") pod \"maas-api-revoke-sub-fa1493c0-4vgv6\" (UID: \"482c7875-59c8-4eed-a756-016214ea4127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1493c0-4vgv6" Oct 30 21:36:52.464723 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.464700 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482c7875-59c8-4eed-a756-016214ea4127-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa1493c0-4vgv6\" (UID: \"482c7875-59c8-4eed-a756-016214ea4127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1493c0-4vgv6" Oct 30 21:36:52.472183 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.471875 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1493c0-4vgv6" Oct 30 21:36:52.825096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:52.825061 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa1493c0-4vgv6"] Oct 30 21:36:52.826726 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:36:52.826699 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482c7875_59c8_4eed_a756_016214ea4127.slice/crio-add289581ed88f6c80009838ea5242869e96ee5c23569c809025ee62440cf077 WatchSource:0}: Error finding container add289581ed88f6c80009838ea5242869e96ee5c23569c809025ee62440cf077: Status 404 returned error can't find the container with id add289581ed88f6c80009838ea5242869e96ee5c23569c809025ee62440cf077 Oct 30 21:36:53.073576 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:53.073483 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1493c0-4vgv6" event={"ID":"482c7875-59c8-4eed-a756-016214ea4127","Type":"ContainerStarted","Data":"add289581ed88f6c80009838ea5242869e96ee5c23569c809025ee62440cf077"} Oct 30 21:36:54.086110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:54.085128 4926 generic.go:356] "Generic (PLEG): container finished" podID="482c7875-59c8-4eed-a756-016214ea4127" containerID="fe436d04d360d00adf7a1e2505c286f9282703da97edf38f7242a59c2f5a3b7e" exitCode=0 Oct 30 21:36:54.086110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:54.085175 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1493c0-4vgv6" event={"ID":"482c7875-59c8-4eed-a756-016214ea4127","Type":"ContainerDied","Data":"fe436d04d360d00adf7a1e2505c286f9282703da97edf38f7242a59c2f5a3b7e"} Oct 30 21:36:55.330631 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:55.330604 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1493c0-4vgv6" Oct 30 21:36:55.480438 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:55.480389 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/482c7875-59c8-4eed-a756-016214ea4127-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482c7875-59c8-4eed-a756-016214ea4127-maas-api-openshift-ca\") pod \"482c7875-59c8-4eed-a756-016214ea4127\" (UID: \"482c7875-59c8-4eed-a756-016214ea4127\") " Oct 30 21:36:55.480627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:55.480555 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/482c7875-59c8-4eed-a756-016214ea4127-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482c7875-59c8-4eed-a756-016214ea4127-maas-api-xks-ca\") pod \"482c7875-59c8-4eed-a756-016214ea4127\" (UID: \"482c7875-59c8-4eed-a756-016214ea4127\") " Oct 30 21:36:55.480627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:55.480617 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/482c7875-59c8-4eed-a756-016214ea4127-kube-api-access-5pwm6\" (UniqueName: \"kubernetes.io/projected/482c7875-59c8-4eed-a756-016214ea4127-kube-api-access-5pwm6\") pod \"482c7875-59c8-4eed-a756-016214ea4127\" (UID: \"482c7875-59c8-4eed-a756-016214ea4127\") " Oct 30 21:36:55.480830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:55.480801 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/482c7875-59c8-4eed-a756-016214ea4127-maas-api-openshift-ca" pod "482c7875-59c8-4eed-a756-016214ea4127" (UID: "482c7875-59c8-4eed-a756-016214ea4127"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:55.481368 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:55.481341 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482c7875-59c8-4eed-a756-016214ea4127-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:55.482783 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:55.482750 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/482c7875-59c8-4eed-a756-016214ea4127-maas-api-xks-ca" pod "482c7875-59c8-4eed-a756-016214ea4127" (UID: "482c7875-59c8-4eed-a756-016214ea4127"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:36:55.482898 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:55.482838 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482c7875-59c8-4eed-a756-016214ea4127-kube-api-access-5pwm6" pod "482c7875-59c8-4eed-a756-016214ea4127" (UID: "482c7875-59c8-4eed-a756-016214ea4127"). InnerVolumeSpecName "kube-api-access-5pwm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:55.582491 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:55.582452 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482c7875-59c8-4eed-a756-016214ea4127-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:55.582491 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:55.582485 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pwm6\" (UniqueName: \"kubernetes.io/projected/482c7875-59c8-4eed-a756-016214ea4127-kube-api-access-5pwm6\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:56.110227 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:56.110186 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1493c0-4vgv6" Oct 30 21:36:56.110423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:56.110301 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa1493c0-4vgv6" event={"ID":"482c7875-59c8-4eed-a756-016214ea4127","Type":"ContainerDied","Data":"add289581ed88f6c80009838ea5242869e96ee5c23569c809025ee62440cf077"} Oct 30 21:36:56.110423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:56.110357 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="add289581ed88f6c80009838ea5242869e96ee5c23569c809025ee62440cf077" Oct 30 21:36:57.980989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:57.980952 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr"] Oct 30 21:36:58.005290 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:58.003175 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" podUID="c79ad431-18a6-4301-bb75-db4a80b01755" containerName="maas-api" containerID="cri-o://6a9c97a15f2d1654bf43c88a22a8052f1c25204f0794d0f37e36b3667678be65" gracePeriod=30 Oct 30 21:36:58.129846 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:58.129807 4926 generic.go:356] "Generic (PLEG): container finished" podID="c79ad431-18a6-4301-bb75-db4a80b01755" containerID="6a9c97a15f2d1654bf43c88a22a8052f1c25204f0794d0f37e36b3667678be65" exitCode=0 Oct 30 21:36:58.129988 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:58.129885 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" event={"ID":"c79ad431-18a6-4301-bb75-db4a80b01755","Type":"ContainerDied","Data":"6a9c97a15f2d1654bf43c88a22a8052f1c25204f0794d0f37e36b3667678be65"} Oct 30 21:36:58.635806 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:58.635776 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" Oct 30 21:36:58.710437 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:58.708979 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c79ad431-18a6-4301-bb75-db4a80b01755-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c79ad431-18a6-4301-bb75-db4a80b01755-maas-api-tls\") pod \"c79ad431-18a6-4301-bb75-db4a80b01755\" (UID: \"c79ad431-18a6-4301-bb75-db4a80b01755\") " Oct 30 21:36:58.710437 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:58.709109 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c79ad431-18a6-4301-bb75-db4a80b01755-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c79ad431-18a6-4301-bb75-db4a80b01755-openshift-service-ca\") pod \"c79ad431-18a6-4301-bb75-db4a80b01755\" (UID: \"c79ad431-18a6-4301-bb75-db4a80b01755\") " Oct 30 21:36:58.710437 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:58.709234 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c79ad431-18a6-4301-bb75-db4a80b01755-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c79ad431-18a6-4301-bb75-db4a80b01755-metrics-certs\") pod \"c79ad431-18a6-4301-bb75-db4a80b01755\" (UID: \"c79ad431-18a6-4301-bb75-db4a80b01755\") " Oct 30 21:36:58.710437 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:58.709267 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c79ad431-18a6-4301-bb75-db4a80b01755-kube-api-access-ffrrc\" (UniqueName: \"kubernetes.io/projected/c79ad431-18a6-4301-bb75-db4a80b01755-kube-api-access-ffrrc\") pod \"c79ad431-18a6-4301-bb75-db4a80b01755\" (UID: \"c79ad431-18a6-4301-bb75-db4a80b01755\") " Oct 30 21:36:58.711207 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:58.711172 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c79ad431-18a6-4301-bb75-db4a80b01755-openshift-service-ca" pod "c79ad431-18a6-4301-bb75-db4a80b01755" (UID: "c79ad431-18a6-4301-bb75-db4a80b01755"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:36:58.720278 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:58.720185 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c79ad431-18a6-4301-bb75-db4a80b01755-maas-api-tls" pod "c79ad431-18a6-4301-bb75-db4a80b01755" (UID: "c79ad431-18a6-4301-bb75-db4a80b01755"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:36:58.720278 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:58.720218 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c79ad431-18a6-4301-bb75-db4a80b01755-kube-api-access-ffrrc" pod "c79ad431-18a6-4301-bb75-db4a80b01755" (UID: "c79ad431-18a6-4301-bb75-db4a80b01755"). InnerVolumeSpecName "kube-api-access-ffrrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:36:58.721444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:58.721416 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c79ad431-18a6-4301-bb75-db4a80b01755-metrics-certs" pod "c79ad431-18a6-4301-bb75-db4a80b01755" (UID: "c79ad431-18a6-4301-bb75-db4a80b01755"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:36:58.811637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:58.811599 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c79ad431-18a6-4301-bb75-db4a80b01755-maas-api-tls\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:58.811637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:58.811637 4926 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c79ad431-18a6-4301-bb75-db4a80b01755-openshift-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:58.811637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:58.811648 4926 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c79ad431-18a6-4301-bb75-db4a80b01755-metrics-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:58.811901 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:58.811657 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffrrc\" (UniqueName: \"kubernetes.io/projected/c79ad431-18a6-4301-bb75-db4a80b01755-kube-api-access-ffrrc\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:36:58.894712 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:58.894513 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6"] Oct 30 21:36:58.901886 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:58.901785 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5145c-sz5j6"] Oct 30 21:36:59.017959 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:59.017923 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28246ca0-71ac-4f01-bc42-52b5afbedc3a" path="/var/lib/kubelet/pods/28246ca0-71ac-4f01-bc42-52b5afbedc3a/volumes" Oct 30 21:36:59.139910 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:59.139882 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" Oct 30 21:36:59.140146 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:59.139919 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr" event={"ID":"c79ad431-18a6-4301-bb75-db4a80b01755","Type":"ContainerDied","Data":"7188e5e7b7b493aea9aa200a49594749614b7f65057f0b6a2918fb647dd066db"} Oct 30 21:36:59.140146 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:59.139987 4926 scope.go:122] "RemoveContainer" containerID="6a9c97a15f2d1654bf43c88a22a8052f1c25204f0794d0f37e36b3667678be65" Oct 30 21:36:59.203358 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:59.203308 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr"] Oct 30 21:36:59.209063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:36:59.209015 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c5145c-76d868b89c-xbssr"] Oct 30 21:37:01.020562 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:01.020529 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c79ad431-18a6-4301-bb75-db4a80b01755" path="/var/lib/kubelet/pods/c79ad431-18a6-4301-bb75-db4a80b01755/volumes" Oct 30 21:37:04.232437 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:04.232391 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq"] Oct 30 21:37:04.243259 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:04.243229 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79c24a-5jxrq"] Oct 30 21:37:05.020011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:05.019969 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6ef610a-9e6f-47cc-9e18-7406909d5df1" path="/var/lib/kubelet/pods/d6ef610a-9e6f-47cc-9e18-7406909d5df1/volumes" Oct 30 21:37:06.335987 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:06.335941 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" podUID="9c2bb6ce-be54-4671-a234-759effcd695c" containerName="main" containerID="cri-o://5788041a8b152acfb1674a710814271f42981f996f1638310bad74402cc3508e" gracePeriod=45 Oct 30 21:37:06.758296 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:06.758260 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r"] Oct 30 21:37:06.758542 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:06.758514 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" podUID="2eea6ad7-9d9b-4205-89d2-af24991331de" containerName="istio-proxy" containerID="cri-o://7191f18d5a586ae57a6f22b17224e3217055ff9cffd5606b933ca51a4ca0d391" gracePeriod=30 Oct 30 21:37:06.809502 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:06.809465 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:37:06.979120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:06.976519 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-dshm\") pod \"9c2bb6ce-be54-4671-a234-759effcd695c\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " Oct 30 21:37:06.979120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:06.976577 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-home\" (UniqueName: \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-home\") pod \"9c2bb6ce-be54-4671-a234-759effcd695c\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " Oct 30 21:37:06.979120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:06.976722 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c2bb6ce-be54-4671-a234-759effcd695c-tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2bb6ce-be54-4671-a234-759effcd695c-tls-certs\") pod \"9c2bb6ce-be54-4671-a234-759effcd695c\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " Oct 30 21:37:06.979120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:06.976760 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c2bb6ce-be54-4671-a234-759effcd695c-kube-api-access-mthq9\" (UniqueName: \"kubernetes.io/projected/9c2bb6ce-be54-4671-a234-759effcd695c-kube-api-access-mthq9\") pod \"9c2bb6ce-be54-4671-a234-759effcd695c\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " Oct 30 21:37:06.979120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:06.976983 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-model-cache\") pod \"9c2bb6ce-be54-4671-a234-759effcd695c\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " Oct 30 21:37:06.979120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:06.977064 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-tmp-dir\") pod \"9c2bb6ce-be54-4671-a234-759effcd695c\" (UID: \"9c2bb6ce-be54-4671-a234-759effcd695c\") " Oct 30 21:37:06.980119 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:06.979731 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-tmp-dir" pod "9c2bb6ce-be54-4671-a234-759effcd695c" (UID: "9c2bb6ce-be54-4671-a234-759effcd695c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:06.981398 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:06.981373 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-model-cache" pod "9c2bb6ce-be54-4671-a234-759effcd695c" (UID: "9c2bb6ce-be54-4671-a234-759effcd695c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:06.981604 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:06.981583 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-home" pod "9c2bb6ce-be54-4671-a234-759effcd695c" (UID: "9c2bb6ce-be54-4671-a234-759effcd695c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:07.006649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.005473 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c2bb6ce-be54-4671-a234-759effcd695c-tls-certs" pod "9c2bb6ce-be54-4671-a234-759effcd695c" (UID: "9c2bb6ce-be54-4671-a234-759effcd695c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:07.006649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.006124 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2bb6ce-be54-4671-a234-759effcd695c-kube-api-access-mthq9" pod "9c2bb6ce-be54-4671-a234-759effcd695c" (UID: "9c2bb6ce-be54-4671-a234-759effcd695c"). InnerVolumeSpecName "kube-api-access-mthq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:07.006649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.006577 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-dshm" pod "9c2bb6ce-be54-4671-a234-759effcd695c" (UID: "9c2bb6ce-be54-4671-a234-759effcd695c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:07.088293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.079464 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:07.088293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.079496 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:07.088293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.079509 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:07.088293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.079521 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c2bb6ce-be54-4671-a234-759effcd695c-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:07.088293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.079532 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c2bb6ce-be54-4671-a234-759effcd695c-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:07.088293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.079545 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mthq9\" (UniqueName: \"kubernetes.io/projected/9c2bb6ce-be54-4671-a234-759effcd695c-kube-api-access-mthq9\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:07.091568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.090755 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc"] Oct 30 21:37:07.092529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.092506 4926 cpu_manager.go:414] "containerMap: removing container" podUID="482c7875-59c8-4eed-a756-016214ea4127" containerName="revoke-keys" Oct 30 21:37:07.092529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.092527 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="482c7875-59c8-4eed-a756-016214ea4127" containerName="revoke-keys" Oct 30 21:37:07.092729 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.092541 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c79ad431-18a6-4301-bb75-db4a80b01755" containerName="maas-api" Oct 30 21:37:07.092729 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.092547 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c79ad431-18a6-4301-bb75-db4a80b01755" containerName="maas-api" Oct 30 21:37:07.092729 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.092577 4926 cpu_manager.go:414] "containerMap: removing container" podUID="9c2bb6ce-be54-4671-a234-759effcd695c" containerName="main" Oct 30 21:37:07.092729 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.092582 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c2bb6ce-be54-4671-a234-759effcd695c" containerName="main" Oct 30 21:37:07.092729 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.092722 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c2bb6ce-be54-4671-a234-759effcd695c" containerName="main" Oct 30 21:37:07.092997 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.092739 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="482c7875-59c8-4eed-a756-016214ea4127" containerName="revoke-keys" Oct 30 21:37:07.092997 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.092747 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c79ad431-18a6-4301-bb75-db4a80b01755" containerName="maas-api" Oct 30 21:37:07.096568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.096542 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc" Oct 30 21:37:07.112906 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.112229 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:37:07.112906 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.112363 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:37:07.117893 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.117867 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc"] Oct 30 21:37:07.180580 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.180543 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b496030-08c9-4bec-9ded-8d9b40f2f66b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4cd1c85-s5xvc\" (UID: \"3b496030-08c9-4bec-9ded-8d9b40f2f66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc" Oct 30 21:37:07.180793 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.180691 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtm6r\" (UniqueName: \"kubernetes.io/projected/3b496030-08c9-4bec-9ded-8d9b40f2f66b-kube-api-access-jtm6r\") pod \"maas-api-revoke-sub-d4cd1c85-s5xvc\" (UID: \"3b496030-08c9-4bec-9ded-8d9b40f2f66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc" Oct 30 21:37:07.180861 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.180800 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b496030-08c9-4bec-9ded-8d9b40f2f66b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4cd1c85-s5xvc\" (UID: \"3b496030-08c9-4bec-9ded-8d9b40f2f66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc" Oct 30 21:37:07.218170 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.218128 4926 generic.go:356] "Generic (PLEG): container finished" podID="9c2bb6ce-be54-4671-a234-759effcd695c" containerID="5788041a8b152acfb1674a710814271f42981f996f1638310bad74402cc3508e" exitCode=0 Oct 30 21:37:07.218351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.218173 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" Oct 30 21:37:07.218351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.218208 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" event={"ID":"9c2bb6ce-be54-4671-a234-759effcd695c","Type":"ContainerDied","Data":"5788041a8b152acfb1674a710814271f42981f996f1638310bad74402cc3508e"} Oct 30 21:37:07.218351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.218259 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s" event={"ID":"9c2bb6ce-be54-4671-a234-759effcd695c","Type":"ContainerDied","Data":"49b0ab9cbf0670f90749cd63e6232f2ce2f39a269000a12f3131d311b5c9d08e"} Oct 30 21:37:07.218351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.218278 4926 scope.go:122] "RemoveContainer" containerID="5788041a8b152acfb1674a710814271f42981f996f1638310bad74402cc3508e" Oct 30 21:37:07.229541 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.229520 4926 scope.go:122] "RemoveContainer" containerID="5788041a8b152acfb1674a710814271f42981f996f1638310bad74402cc3508e" Oct 30 21:37:07.229843 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:37:07.229823 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5788041a8b152acfb1674a710814271f42981f996f1638310bad74402cc3508e\": container with ID starting with 5788041a8b152acfb1674a710814271f42981f996f1638310bad74402cc3508e not found: ID does not exist" containerID="5788041a8b152acfb1674a710814271f42981f996f1638310bad74402cc3508e" Oct 30 21:37:07.229921 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.229857 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5788041a8b152acfb1674a710814271f42981f996f1638310bad74402cc3508e"} err="failed to get container status \"5788041a8b152acfb1674a710814271f42981f996f1638310bad74402cc3508e\": rpc error: code = NotFound desc = could not find container \"5788041a8b152acfb1674a710814271f42981f996f1638310bad74402cc3508e\": container with ID starting with 5788041a8b152acfb1674a710814271f42981f996f1638310bad74402cc3508e not found: ID does not exist" Oct 30 21:37:07.269497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.269455 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s"] Oct 30 21:37:07.274766 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.274730 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-79e911/test-model-79e911-kserve-7479c5cd75-8kw7s"] Oct 30 21:37:07.281734 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.281694 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b496030-08c9-4bec-9ded-8d9b40f2f66b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4cd1c85-s5xvc\" (UID: \"3b496030-08c9-4bec-9ded-8d9b40f2f66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc" Oct 30 21:37:07.281927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.281854 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b496030-08c9-4bec-9ded-8d9b40f2f66b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4cd1c85-s5xvc\" (UID: \"3b496030-08c9-4bec-9ded-8d9b40f2f66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc" Oct 30 21:37:07.282008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.281947 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtm6r\" (UniqueName: \"kubernetes.io/projected/3b496030-08c9-4bec-9ded-8d9b40f2f66b-kube-api-access-jtm6r\") pod \"maas-api-revoke-sub-d4cd1c85-s5xvc\" (UID: \"3b496030-08c9-4bec-9ded-8d9b40f2f66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc" Oct 30 21:37:07.282718 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.282686 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b496030-08c9-4bec-9ded-8d9b40f2f66b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4cd1c85-s5xvc\" (UID: \"3b496030-08c9-4bec-9ded-8d9b40f2f66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc" Oct 30 21:37:07.284792 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.284766 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b496030-08c9-4bec-9ded-8d9b40f2f66b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4cd1c85-s5xvc\" (UID: \"3b496030-08c9-4bec-9ded-8d9b40f2f66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc" Oct 30 21:37:07.302817 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.302784 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtm6r\" (UniqueName: \"kubernetes.io/projected/3b496030-08c9-4bec-9ded-8d9b40f2f66b-kube-api-access-jtm6r\") pod \"maas-api-revoke-sub-d4cd1c85-s5xvc\" (UID: \"3b496030-08c9-4bec-9ded-8d9b40f2f66b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc" Oct 30 21:37:07.310068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.310015 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm"] Oct 30 21:37:07.315855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.315837 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm" Oct 30 21:37:07.333842 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.333818 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm"] Oct 30 21:37:07.384468 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.384421 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051a4c83-ae7a-47e1-988d-70724bc221ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm\" (UID: \"051a4c83-ae7a-47e1-988d-70724bc221ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm" Oct 30 21:37:07.384977 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.384621 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8sbx\" (UniqueName: \"kubernetes.io/projected/051a4c83-ae7a-47e1-988d-70724bc221ec-kube-api-access-t8sbx\") pod \"maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm\" (UID: \"051a4c83-ae7a-47e1-988d-70724bc221ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm" Oct 30 21:37:07.384977 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.384848 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051a4c83-ae7a-47e1-988d-70724bc221ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm\" (UID: \"051a4c83-ae7a-47e1-988d-70724bc221ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm" Oct 30 21:37:07.419223 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.419186 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc" Oct 30 21:37:07.486580 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.486200 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8sbx\" (UniqueName: \"kubernetes.io/projected/051a4c83-ae7a-47e1-988d-70724bc221ec-kube-api-access-t8sbx\") pod \"maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm\" (UID: \"051a4c83-ae7a-47e1-988d-70724bc221ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm" Oct 30 21:37:07.486580 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.486392 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051a4c83-ae7a-47e1-988d-70724bc221ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm\" (UID: \"051a4c83-ae7a-47e1-988d-70724bc221ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm" Oct 30 21:37:07.486580 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.486469 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051a4c83-ae7a-47e1-988d-70724bc221ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm\" (UID: \"051a4c83-ae7a-47e1-988d-70724bc221ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm" Oct 30 21:37:07.487380 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.487354 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051a4c83-ae7a-47e1-988d-70724bc221ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm\" (UID: \"051a4c83-ae7a-47e1-988d-70724bc221ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm" Oct 30 21:37:07.514848 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.514811 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051a4c83-ae7a-47e1-988d-70724bc221ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm\" (UID: \"051a4c83-ae7a-47e1-988d-70724bc221ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm" Oct 30 21:37:07.521844 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.521717 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8sbx\" (UniqueName: \"kubernetes.io/projected/051a4c83-ae7a-47e1-988d-70724bc221ec-kube-api-access-t8sbx\") pod \"maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm\" (UID: \"051a4c83-ae7a-47e1-988d-70724bc221ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm" Oct 30 21:37:07.524095 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:37:07.523218 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:37:07.631120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.631077 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm" Oct 30 21:37:07.728953 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:37:07.728919 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b496030_08c9_4bec_9ded_8d9b40f2f66b.slice/crio-c83fe6589f587cd425e0037653e40823ce623a5a8c832b6db71b8c84d5098436 WatchSource:0}: Error finding container c83fe6589f587cd425e0037653e40823ce623a5a8c832b6db71b8c84d5098436: Status 404 returned error can't find the container with id c83fe6589f587cd425e0037653e40823ce623a5a8c832b6db71b8c84d5098436 Oct 30 21:37:07.729509 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.729481 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc"] Oct 30 21:37:07.946393 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:07.946364 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm"] Oct 30 21:37:07.969632 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:37:07.969593 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051a4c83_ae7a_47e1_988d_70724bc221ec.slice/crio-09e637910ad4d67176c470cb43c76b6c2fc40a974f2051d94fa9f6ede2f8aeca WatchSource:0}: Error finding container 09e637910ad4d67176c470cb43c76b6c2fc40a974f2051d94fa9f6ede2f8aeca: Status 404 returned error can't find the container with id 09e637910ad4d67176c470cb43c76b6c2fc40a974f2051d94fa9f6ede2f8aeca Oct 30 21:37:08.229825 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:08.229786 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm" event={"ID":"051a4c83-ae7a-47e1-988d-70724bc221ec","Type":"ContainerStarted","Data":"09e637910ad4d67176c470cb43c76b6c2fc40a974f2051d94fa9f6ede2f8aeca"} Oct 30 21:37:08.231589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:08.231555 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc" event={"ID":"3b496030-08c9-4bec-9ded-8d9b40f2f66b","Type":"ContainerStarted","Data":"d0f39364d744f0c06c0aff5244391ae7d4b86af5d9de25d2cc855a1613a314f6"} Oct 30 21:37:08.231702 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:08.231605 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc" event={"ID":"3b496030-08c9-4bec-9ded-8d9b40f2f66b","Type":"ContainerStarted","Data":"c83fe6589f587cd425e0037653e40823ce623a5a8c832b6db71b8c84d5098436"} Oct 30 21:37:08.278476 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:08.278357 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc" podStartSLOduration=1.278342163 podStartE2EDuration="1.278342163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:08.273878273 +0000 UTC m=+2215.394952021" watchObservedRunningTime="2026-10-30 21:37:08.278342163 +0000 UTC m=+2215.399415950" Oct 30 21:37:09.019403 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:09.019323 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c2bb6ce-be54-4671-a234-759effcd695c" path="/var/lib/kubelet/pods/9c2bb6ce-be54-4671-a234-759effcd695c/volumes" Oct 30 21:37:09.241022 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:09.240983 4926 generic.go:356] "Generic (PLEG): container finished" podID="3b496030-08c9-4bec-9ded-8d9b40f2f66b" containerID="d0f39364d744f0c06c0aff5244391ae7d4b86af5d9de25d2cc855a1613a314f6" exitCode=0 Oct 30 21:37:09.241246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:09.241072 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc" event={"ID":"3b496030-08c9-4bec-9ded-8d9b40f2f66b","Type":"ContainerDied","Data":"d0f39364d744f0c06c0aff5244391ae7d4b86af5d9de25d2cc855a1613a314f6"} Oct 30 21:37:09.242990 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:09.242965 4926 generic.go:356] "Generic (PLEG): container finished" podID="051a4c83-ae7a-47e1-988d-70724bc221ec" containerID="6ed84bda7b7d5397f6823381460eb17f1d4413e8f17d0f798e655ea1a423fd76" exitCode=0 Oct 30 21:37:09.243141 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:09.243015 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm" event={"ID":"051a4c83-ae7a-47e1-988d-70724bc221ec","Type":"ContainerDied","Data":"6ed84bda7b7d5397f6823381460eb17f1d4413e8f17d0f798e655ea1a423fd76"} Oct 30 21:37:10.443127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:10.443096 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm" Oct 30 21:37:10.470443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:10.470415 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc" Oct 30 21:37:10.539346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:10.539262 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b496030-08c9-4bec-9ded-8d9b40f2f66b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b496030-08c9-4bec-9ded-8d9b40f2f66b-maas-api-openshift-ca\") pod \"3b496030-08c9-4bec-9ded-8d9b40f2f66b\" (UID: \"3b496030-08c9-4bec-9ded-8d9b40f2f66b\") " Oct 30 21:37:10.539346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:10.539321 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b496030-08c9-4bec-9ded-8d9b40f2f66b-kube-api-access-jtm6r\" (UniqueName: \"kubernetes.io/projected/3b496030-08c9-4bec-9ded-8d9b40f2f66b-kube-api-access-jtm6r\") pod \"3b496030-08c9-4bec-9ded-8d9b40f2f66b\" (UID: \"3b496030-08c9-4bec-9ded-8d9b40f2f66b\") " Oct 30 21:37:10.539531 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:10.539376 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/051a4c83-ae7a-47e1-988d-70724bc221ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051a4c83-ae7a-47e1-988d-70724bc221ec-maas-api-openshift-ca\") pod \"051a4c83-ae7a-47e1-988d-70724bc221ec\" (UID: \"051a4c83-ae7a-47e1-988d-70724bc221ec\") " Oct 30 21:37:10.539704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:10.539680 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b496030-08c9-4bec-9ded-8d9b40f2f66b-maas-api-openshift-ca" pod "3b496030-08c9-4bec-9ded-8d9b40f2f66b" (UID: "3b496030-08c9-4bec-9ded-8d9b40f2f66b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:10.539764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:10.539718 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/051a4c83-ae7a-47e1-988d-70724bc221ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051a4c83-ae7a-47e1-988d-70724bc221ec-maas-api-xks-ca\") pod \"051a4c83-ae7a-47e1-988d-70724bc221ec\" (UID: \"051a4c83-ae7a-47e1-988d-70724bc221ec\") " Oct 30 21:37:10.539764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:10.539744 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/051a4c83-ae7a-47e1-988d-70724bc221ec-kube-api-access-t8sbx\" (UniqueName: \"kubernetes.io/projected/051a4c83-ae7a-47e1-988d-70724bc221ec-kube-api-access-t8sbx\") pod \"051a4c83-ae7a-47e1-988d-70724bc221ec\" (UID: \"051a4c83-ae7a-47e1-988d-70724bc221ec\") " Oct 30 21:37:10.539876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:10.539758 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/051a4c83-ae7a-47e1-988d-70724bc221ec-maas-api-openshift-ca" pod "051a4c83-ae7a-47e1-988d-70724bc221ec" (UID: "051a4c83-ae7a-47e1-988d-70724bc221ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:10.539876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:10.539764 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b496030-08c9-4bec-9ded-8d9b40f2f66b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b496030-08c9-4bec-9ded-8d9b40f2f66b-maas-api-xks-ca\") pod \"3b496030-08c9-4bec-9ded-8d9b40f2f66b\" (UID: \"3b496030-08c9-4bec-9ded-8d9b40f2f66b\") " Oct 30 21:37:10.540450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:10.540426 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b496030-08c9-4bec-9ded-8d9b40f2f66b-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:10.540450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:10.540453 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/051a4c83-ae7a-47e1-988d-70724bc221ec-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:10.542238 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:10.542208 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/051a4c83-ae7a-47e1-988d-70724bc221ec-kube-api-access-t8sbx" pod "051a4c83-ae7a-47e1-988d-70724bc221ec" (UID: "051a4c83-ae7a-47e1-988d-70724bc221ec"). InnerVolumeSpecName "kube-api-access-t8sbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:10.542502 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:10.542469 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/051a4c83-ae7a-47e1-988d-70724bc221ec-maas-api-xks-ca" pod "051a4c83-ae7a-47e1-988d-70724bc221ec" (UID: "051a4c83-ae7a-47e1-988d-70724bc221ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:10.542607 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:10.542570 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b496030-08c9-4bec-9ded-8d9b40f2f66b-maas-api-xks-ca" pod "3b496030-08c9-4bec-9ded-8d9b40f2f66b" (UID: "3b496030-08c9-4bec-9ded-8d9b40f2f66b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:10.542687 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:10.542615 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b496030-08c9-4bec-9ded-8d9b40f2f66b-kube-api-access-jtm6r" pod "3b496030-08c9-4bec-9ded-8d9b40f2f66b" (UID: "3b496030-08c9-4bec-9ded-8d9b40f2f66b"). InnerVolumeSpecName "kube-api-access-jtm6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:10.641821 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:10.641781 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/051a4c83-ae7a-47e1-988d-70724bc221ec-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:10.641821 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:10.641816 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8sbx\" (UniqueName: \"kubernetes.io/projected/051a4c83-ae7a-47e1-988d-70724bc221ec-kube-api-access-t8sbx\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:10.642097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:10.641833 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b496030-08c9-4bec-9ded-8d9b40f2f66b-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:10.642097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:10.641846 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtm6r\" (UniqueName: \"kubernetes.io/projected/3b496030-08c9-4bec-9ded-8d9b40f2f66b-kube-api-access-jtm6r\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:11.214057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.213066 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc"] Oct 30 21:37:11.214057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.213274 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" podUID="84b972f9-f7bd-4895-9863-dbf951a09e00" containerName="istio-proxy" containerID="cri-o://1f6c8e587e481b029dd7af5a4d048f2d700920324b7ec3fea6d5be1a7688f540" gracePeriod=30 Oct 30 21:37:11.264967 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.264920 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm" Oct 30 21:37:11.265184 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.264924 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm" event={"ID":"051a4c83-ae7a-47e1-988d-70724bc221ec","Type":"ContainerDied","Data":"09e637910ad4d67176c470cb43c76b6c2fc40a974f2051d94fa9f6ede2f8aeca"} Oct 30 21:37:11.265184 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.265113 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09e637910ad4d67176c470cb43c76b6c2fc40a974f2051d94fa9f6ede2f8aeca" Oct 30 21:37:11.269013 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.268976 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc" event={"ID":"3b496030-08c9-4bec-9ded-8d9b40f2f66b","Type":"ContainerDied","Data":"c83fe6589f587cd425e0037653e40823ce623a5a8c832b6db71b8c84d5098436"} Oct 30 21:37:11.269205 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.269069 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c83fe6589f587cd425e0037653e40823ce623a5a8c832b6db71b8c84d5098436" Oct 30 21:37:11.269205 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.269181 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc" Oct 30 21:37:11.729705 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.729665 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3b28bc-79422"] Oct 30 21:37:11.730361 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.730343 4926 cpu_manager.go:414] "containerMap: removing container" podUID="051a4c83-ae7a-47e1-988d-70724bc221ec" containerName="revoke-keys" Oct 30 21:37:11.730409 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.730362 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="051a4c83-ae7a-47e1-988d-70724bc221ec" containerName="revoke-keys" Oct 30 21:37:11.730456 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.730400 4926 cpu_manager.go:414] "containerMap: removing container" podUID="3b496030-08c9-4bec-9ded-8d9b40f2f66b" containerName="revoke-keys" Oct 30 21:37:11.730456 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.730423 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b496030-08c9-4bec-9ded-8d9b40f2f66b" containerName="revoke-keys" Oct 30 21:37:11.730530 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.730520 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b496030-08c9-4bec-9ded-8d9b40f2f66b" containerName="revoke-keys" Oct 30 21:37:11.730570 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.730536 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="051a4c83-ae7a-47e1-988d-70724bc221ec" containerName="revoke-keys" Oct 30 21:37:11.734492 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.734466 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3b28bc-79422" Oct 30 21:37:11.751669 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.751639 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3b28bc-79422"] Oct 30 21:37:11.775270 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.775232 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:37:11.775489 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.775370 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:37:11.865885 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.865846 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3b28bc-79422\" (UID: \"c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3b28bc-79422" Oct 30 21:37:11.866121 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.866086 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6wtf\" (UniqueName: \"kubernetes.io/projected/c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0-kube-api-access-z6wtf\") pod \"maas-api-revoke-keys-e2e-inf-b-3b28bc-79422\" (UID: \"c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3b28bc-79422" Oct 30 21:37:11.866218 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.866167 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3b28bc-79422\" (UID: \"c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3b28bc-79422" Oct 30 21:37:11.967850 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.967803 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3b28bc-79422\" (UID: \"c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3b28bc-79422" Oct 30 21:37:11.968083 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.967916 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6wtf\" (UniqueName: \"kubernetes.io/projected/c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0-kube-api-access-z6wtf\") pod \"maas-api-revoke-keys-e2e-inf-b-3b28bc-79422\" (UID: \"c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3b28bc-79422" Oct 30 21:37:11.968083 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.967945 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3b28bc-79422\" (UID: \"c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3b28bc-79422" Oct 30 21:37:11.968771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.968740 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3b28bc-79422\" (UID: \"c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3b28bc-79422" Oct 30 21:37:11.970493 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:11.970426 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3b28bc-79422\" (UID: \"c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3b28bc-79422" Oct 30 21:37:12.004520 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.004436 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6wtf\" (UniqueName: \"kubernetes.io/projected/c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0-kube-api-access-z6wtf\") pod \"maas-api-revoke-keys-e2e-inf-b-3b28bc-79422\" (UID: \"c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3b28bc-79422" Oct 30 21:37:12.047311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.047281 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:37:12.051020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.050990 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3b28bc-79422" Oct 30 21:37:12.171364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.170789 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-envoy\") pod \"2eea6ad7-9d9b-4205-89d2-af24991331de\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " Oct 30 21:37:12.171364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.170841 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-data\") pod \"2eea6ad7-9d9b-4205-89d2-af24991331de\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " Oct 30 21:37:12.171364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.170918 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-credential-socket\") pod \"2eea6ad7-9d9b-4205-89d2-af24991331de\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " Oct 30 21:37:12.171364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.170967 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-token\" (UniqueName: \"kubernetes.io/projected/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-token\") pod \"2eea6ad7-9d9b-4205-89d2-af24991331de\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " Oct 30 21:37:12.171364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.171094 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-workload-certs\") pod \"2eea6ad7-9d9b-4205-89d2-af24991331de\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " Oct 30 21:37:12.171364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.171142 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eea6ad7-9d9b-4205-89d2-af24991331de-kube-api-access-tk69z\" (UniqueName: \"kubernetes.io/projected/2eea6ad7-9d9b-4205-89d2-af24991331de-kube-api-access-tk69z\") pod \"2eea6ad7-9d9b-4205-89d2-af24991331de\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " Oct 30 21:37:12.171364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.171204 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-podinfo\") pod \"2eea6ad7-9d9b-4205-89d2-af24991331de\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " Oct 30 21:37:12.171364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.171237 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-workload-socket\") pod \"2eea6ad7-9d9b-4205-89d2-af24991331de\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " Oct 30 21:37:12.171952 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.171399 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2eea6ad7-9d9b-4205-89d2-af24991331de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eea6ad7-9d9b-4205-89d2-af24991331de-istiod-ca-cert\") pod \"2eea6ad7-9d9b-4205-89d2-af24991331de\" (UID: \"2eea6ad7-9d9b-4205-89d2-af24991331de\") " Oct 30 21:37:12.172674 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.172572 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eea6ad7-9d9b-4205-89d2-af24991331de-istiod-ca-cert" pod "2eea6ad7-9d9b-4205-89d2-af24991331de" (UID: "2eea6ad7-9d9b-4205-89d2-af24991331de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:12.173782 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.173204 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-data" pod "2eea6ad7-9d9b-4205-89d2-af24991331de" (UID: "2eea6ad7-9d9b-4205-89d2-af24991331de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:12.177064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.176079 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-workload-socket" pod "2eea6ad7-9d9b-4205-89d2-af24991331de" (UID: "2eea6ad7-9d9b-4205-89d2-af24991331de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:12.177064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.176018 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-workload-certs" pod "2eea6ad7-9d9b-4205-89d2-af24991331de" (UID: "2eea6ad7-9d9b-4205-89d2-af24991331de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:12.184615 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.184499 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-envoy" pod "2eea6ad7-9d9b-4205-89d2-af24991331de" (UID: "2eea6ad7-9d9b-4205-89d2-af24991331de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:12.186092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.184764 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-podinfo" pod "2eea6ad7-9d9b-4205-89d2-af24991331de" (UID: "2eea6ad7-9d9b-4205-89d2-af24991331de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:37:12.186092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.184812 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-token" pod "2eea6ad7-9d9b-4205-89d2-af24991331de" (UID: "2eea6ad7-9d9b-4205-89d2-af24991331de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:12.186092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.184814 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-credential-socket" pod "2eea6ad7-9d9b-4205-89d2-af24991331de" (UID: "2eea6ad7-9d9b-4205-89d2-af24991331de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:12.186092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.184920 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eea6ad7-9d9b-4205-89d2-af24991331de-kube-api-access-tk69z" pod "2eea6ad7-9d9b-4205-89d2-af24991331de" (UID: "2eea6ad7-9d9b-4205-89d2-af24991331de"). InnerVolumeSpecName "kube-api-access-tk69z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:12.273072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.272966 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:12.273072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.273000 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:12.273072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.273012 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:12.273072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.273026 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk69z\" (UniqueName: \"kubernetes.io/projected/2eea6ad7-9d9b-4205-89d2-af24991331de-kube-api-access-tk69z\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:12.273072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.273055 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:12.273072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.273068 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:12.273396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.273080 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eea6ad7-9d9b-4205-89d2-af24991331de-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:12.273396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.273091 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:12.273396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.273102 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eea6ad7-9d9b-4205-89d2-af24991331de-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:12.280251 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.280214 4926 generic.go:356] "Generic (PLEG): container finished" podID="2eea6ad7-9d9b-4205-89d2-af24991331de" containerID="7191f18d5a586ae57a6f22b17224e3217055ff9cffd5606b933ca51a4ca0d391" exitCode=0 Oct 30 21:37:12.280450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.280293 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" event={"ID":"2eea6ad7-9d9b-4205-89d2-af24991331de","Type":"ContainerDied","Data":"7191f18d5a586ae57a6f22b17224e3217055ff9cffd5606b933ca51a4ca0d391"} Oct 30 21:37:12.280450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.280373 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" event={"ID":"2eea6ad7-9d9b-4205-89d2-af24991331de","Type":"ContainerDied","Data":"7deb4cb521eb927cb5f7d79a695ceebbf85d9e9d8e92fdcf6fc5fcd0657761bf"} Oct 30 21:37:12.280450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.280388 4926 scope.go:122] "RemoveContainer" containerID="7191f18d5a586ae57a6f22b17224e3217055ff9cffd5606b933ca51a4ca0d391" Oct 30 21:37:12.280580 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.280295 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r" Oct 30 21:37:12.297797 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.297771 4926 scope.go:122] "RemoveContainer" containerID="d43d73aee50bbddfb466a095c47aab5d3f4afef733d0c6c96e6b2c39dc15637d" Oct 30 21:37:12.311794 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.311766 4926 scope.go:122] "RemoveContainer" containerID="7191f18d5a586ae57a6f22b17224e3217055ff9cffd5606b933ca51a4ca0d391" Oct 30 21:37:12.312226 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:37:12.312191 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7191f18d5a586ae57a6f22b17224e3217055ff9cffd5606b933ca51a4ca0d391\": container with ID starting with 7191f18d5a586ae57a6f22b17224e3217055ff9cffd5606b933ca51a4ca0d391 not found: ID does not exist" containerID="7191f18d5a586ae57a6f22b17224e3217055ff9cffd5606b933ca51a4ca0d391" Oct 30 21:37:12.312341 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.312240 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7191f18d5a586ae57a6f22b17224e3217055ff9cffd5606b933ca51a4ca0d391"} err="failed to get container status \"7191f18d5a586ae57a6f22b17224e3217055ff9cffd5606b933ca51a4ca0d391\": rpc error: code = NotFound desc = could not find container \"7191f18d5a586ae57a6f22b17224e3217055ff9cffd5606b933ca51a4ca0d391\": container with ID starting with 7191f18d5a586ae57a6f22b17224e3217055ff9cffd5606b933ca51a4ca0d391 not found: ID does not exist" Oct 30 21:37:12.312341 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.312266 4926 scope.go:122] "RemoveContainer" containerID="d43d73aee50bbddfb466a095c47aab5d3f4afef733d0c6c96e6b2c39dc15637d" Oct 30 21:37:12.312615 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:37:12.312591 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d43d73aee50bbddfb466a095c47aab5d3f4afef733d0c6c96e6b2c39dc15637d\": container with ID starting with d43d73aee50bbddfb466a095c47aab5d3f4afef733d0c6c96e6b2c39dc15637d not found: ID does not exist" containerID="d43d73aee50bbddfb466a095c47aab5d3f4afef733d0c6c96e6b2c39dc15637d" Oct 30 21:37:12.312689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.312623 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d43d73aee50bbddfb466a095c47aab5d3f4afef733d0c6c96e6b2c39dc15637d"} err="failed to get container status \"d43d73aee50bbddfb466a095c47aab5d3f4afef733d0c6c96e6b2c39dc15637d\": rpc error: code = NotFound desc = could not find container \"d43d73aee50bbddfb466a095c47aab5d3f4afef733d0c6c96e6b2c39dc15637d\": container with ID starting with d43d73aee50bbddfb466a095c47aab5d3f4afef733d0c6c96e6b2c39dc15637d not found: ID does not exist" Oct 30 21:37:12.354865 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.354830 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r"] Oct 30 21:37:12.365886 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.365849 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c5145c-openshift-default-6cb9588fc7-j4m8r"] Oct 30 21:37:12.391540 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:37:12.391492 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90d209a_8ec6_4ebc_ac5d_d7be15fea2a0.slice/crio-d4346538f58b7accfcb0bdd9854e202580b74a4ce5cfb0452992083f04defe18 WatchSource:0}: Error finding container d4346538f58b7accfcb0bdd9854e202580b74a4ce5cfb0452992083f04defe18: Status 404 returned error can't find the container with id d4346538f58b7accfcb0bdd9854e202580b74a4ce5cfb0452992083f04defe18 Oct 30 21:37:12.393848 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:12.393820 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3b28bc-79422"] Oct 30 21:37:13.018454 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:13.018417 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eea6ad7-9d9b-4205-89d2-af24991331de" path="/var/lib/kubelet/pods/2eea6ad7-9d9b-4205-89d2-af24991331de/volumes" Oct 30 21:37:13.294911 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:13.294877 4926 generic.go:356] "Generic (PLEG): container finished" podID="c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0" containerID="ebdf2613534404d72ed8510da406901ea6001254ebb63b999b5b697f44d9db7f" exitCode=0 Oct 30 21:37:13.295074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:13.294966 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3b28bc-79422" event={"ID":"c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0","Type":"ContainerDied","Data":"ebdf2613534404d72ed8510da406901ea6001254ebb63b999b5b697f44d9db7f"} Oct 30 21:37:13.295074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:13.295025 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3b28bc-79422" event={"ID":"c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0","Type":"ContainerStarted","Data":"d4346538f58b7accfcb0bdd9854e202580b74a4ce5cfb0452992083f04defe18"} Oct 30 21:37:14.489512 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:14.489482 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3b28bc-79422" Oct 30 21:37:14.606975 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:14.606872 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0-maas-api-xks-ca\") pod \"c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0\" (UID: \"c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0\") " Oct 30 21:37:14.606975 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:14.606930 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0-maas-api-openshift-ca\") pod \"c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0\" (UID: \"c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0\") " Oct 30 21:37:14.607257 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:14.607052 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0-kube-api-access-z6wtf\" (UniqueName: \"kubernetes.io/projected/c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0-kube-api-access-z6wtf\") pod \"c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0\" (UID: \"c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0\") " Oct 30 21:37:14.607388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:14.607358 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0-maas-api-openshift-ca" pod "c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0" (UID: "c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:14.607671 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:14.607651 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:14.609157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:14.609120 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0-maas-api-xks-ca" pod "c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0" (UID: "c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:14.609265 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:14.609173 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0-kube-api-access-z6wtf" pod "c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0" (UID: "c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0"). InnerVolumeSpecName "kube-api-access-z6wtf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:14.709253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:14.709213 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6wtf\" (UniqueName: \"kubernetes.io/projected/c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0-kube-api-access-z6wtf\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:14.709253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:14.709247 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:15.316074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:15.315966 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3b28bc-79422" event={"ID":"c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0","Type":"ContainerDied","Data":"d4346538f58b7accfcb0bdd9854e202580b74a4ce5cfb0452992083f04defe18"} Oct 30 21:37:15.316352 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:15.316078 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4346538f58b7accfcb0bdd9854e202580b74a4ce5cfb0452992083f04defe18" Oct 30 21:37:15.316352 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:15.315978 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3b28bc-79422" Oct 30 21:37:16.519373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.519344 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:37:16.628446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.628407 4926 scope.go:122] "RemoveContainer" containerID="1f6c8e587e481b029dd7af5a4d048f2d700920324b7ec3fea6d5be1a7688f540" Oct 30 21:37:16.634359 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.634334 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-data\") pod \"84b972f9-f7bd-4895-9863-dbf951a09e00\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " Oct 30 21:37:16.634507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.634430 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-token\" (UniqueName: \"kubernetes.io/projected/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-token\") pod \"84b972f9-f7bd-4895-9863-dbf951a09e00\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " Oct 30 21:37:16.634507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.634473 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84b972f9-f7bd-4895-9863-dbf951a09e00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b972f9-f7bd-4895-9863-dbf951a09e00-istiod-ca-cert\") pod \"84b972f9-f7bd-4895-9863-dbf951a09e00\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " Oct 30 21:37:16.634657 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.634629 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-workload-socket\") pod \"84b972f9-f7bd-4895-9863-dbf951a09e00\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " Oct 30 21:37:16.634969 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.634641 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-data" pod "84b972f9-f7bd-4895-9863-dbf951a09e00" (UID: "84b972f9-f7bd-4895-9863-dbf951a09e00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:16.635051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.634839 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84b972f9-f7bd-4895-9863-dbf951a09e00-istiod-ca-cert" pod "84b972f9-f7bd-4895-9863-dbf951a09e00" (UID: "84b972f9-f7bd-4895-9863-dbf951a09e00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:16.635051 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.634890 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-workload-socket" pod "84b972f9-f7bd-4895-9863-dbf951a09e00" (UID: "84b972f9-f7bd-4895-9863-dbf951a09e00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:16.635129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.635087 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-envoy\") pod \"84b972f9-f7bd-4895-9863-dbf951a09e00\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " Oct 30 21:37:16.635174 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.635164 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b972f9-f7bd-4895-9863-dbf951a09e00-kube-api-access-vnv4v\" (UniqueName: \"kubernetes.io/projected/84b972f9-f7bd-4895-9863-dbf951a09e00-kube-api-access-vnv4v\") pod \"84b972f9-f7bd-4895-9863-dbf951a09e00\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " Oct 30 21:37:16.635250 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.635232 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-workload-certs\") pod \"84b972f9-f7bd-4895-9863-dbf951a09e00\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " Oct 30 21:37:16.635328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.635272 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-podinfo\") pod \"84b972f9-f7bd-4895-9863-dbf951a09e00\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " Oct 30 21:37:16.635390 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.635348 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-credential-socket\") pod \"84b972f9-f7bd-4895-9863-dbf951a09e00\" (UID: \"84b972f9-f7bd-4895-9863-dbf951a09e00\") " Oct 30 21:37:16.635996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.635971 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-workload-certs" pod "84b972f9-f7bd-4895-9863-dbf951a09e00" (UID: "84b972f9-f7bd-4895-9863-dbf951a09e00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:16.636190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.636021 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:16.636357 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.636341 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84b972f9-f7bd-4895-9863-dbf951a09e00-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:16.636476 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.636453 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:16.636476 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.636274 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-credential-socket" pod "84b972f9-f7bd-4895-9863-dbf951a09e00" (UID: "84b972f9-f7bd-4895-9863-dbf951a09e00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:16.637555 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.637526 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-envoy" pod "84b972f9-f7bd-4895-9863-dbf951a09e00" (UID: "84b972f9-f7bd-4895-9863-dbf951a09e00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:16.637809 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.637775 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b972f9-f7bd-4895-9863-dbf951a09e00-kube-api-access-vnv4v" pod "84b972f9-f7bd-4895-9863-dbf951a09e00" (UID: "84b972f9-f7bd-4895-9863-dbf951a09e00"). InnerVolumeSpecName "kube-api-access-vnv4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:16.638369 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.638344 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-podinfo" pod "84b972f9-f7bd-4895-9863-dbf951a09e00" (UID: "84b972f9-f7bd-4895-9863-dbf951a09e00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:37:16.647410 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.647377 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-token" pod "84b972f9-f7bd-4895-9863-dbf951a09e00" (UID: "84b972f9-f7bd-4895-9863-dbf951a09e00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:16.737496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.737463 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:16.737496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.737499 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnv4v\" (UniqueName: \"kubernetes.io/projected/84b972f9-f7bd-4895-9863-dbf951a09e00-kube-api-access-vnv4v\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:16.737721 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.737516 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:16.737721 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.737533 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:16.737721 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.737548 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84b972f9-f7bd-4895-9863-dbf951a09e00-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:16.737721 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:16.737562 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84b972f9-f7bd-4895-9863-dbf951a09e00-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:17.333697 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:17.333608 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" event={"ID":"84b972f9-f7bd-4895-9863-dbf951a09e00","Type":"ContainerDied","Data":"1f6c8e587e481b029dd7af5a4d048f2d700920324b7ec3fea6d5be1a7688f540"} Oct 30 21:37:17.333697 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:17.333684 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" event={"ID":"84b972f9-f7bd-4895-9863-dbf951a09e00","Type":"ContainerDied","Data":"96971bb47d9ab3f95d7b4720a19019c649db4f15b0cd821f4a4bdba0abc6bbf6"} Oct 30 21:37:17.333924 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:17.333800 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc" Oct 30 21:37:17.479074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:17.478907 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc"] Oct 30 21:37:17.485129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:17.484922 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-79c24a-openshift-default-6bdc7cfc7f-vwphc"] Oct 30 21:37:17.770748 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:17.770707 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj"] Oct 30 21:37:17.802537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:17.802499 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk"] Oct 30 21:37:18.143209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:18.143120 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g"] Oct 30 21:37:18.149827 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:18.149796 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-290ede-4q96g"] Oct 30 21:37:19.019053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:19.019004 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b972f9-f7bd-4895-9863-dbf951a09e00" path="/var/lib/kubelet/pods/84b972f9-f7bd-4895-9863-dbf951a09e00/volumes" Oct 30 21:37:19.019599 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:19.019579 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd0feb74-1a47-45c7-8493-c420a98692a9" path="/var/lib/kubelet/pods/dd0feb74-1a47-45c7-8493-c420a98692a9/volumes" Oct 30 21:37:22.775056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:22.772201 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj" podUID="7f05e57c-842d-4aac-b454-10ca04014727" containerName="payload-processing" containerID="cri-o://a19cb5d2133186dd977067429408f91404bb6231735c6dc3b243ec0cf3fa4b93" gracePeriod=25 Oct 30 21:37:22.804679 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:22.804630 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk" podUID="23944deb-c056-466d-b79d-d56a257d98df" containerName="payload-pre-processing" containerID="cri-o://2e2c63096b7b3ea96867edb30086c08735468c0ff58091580e06ac1f49b57bee" gracePeriod=25 Oct 30 21:37:23.847064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:23.846411 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj" podUID="7f05e57c-842d-4aac-b454-10ca04014727" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:37:23.847628 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:23.847215 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk" podUID="23944deb-c056-466d-b79d-d56a257d98df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:37:24.013535 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.013486 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f"] Oct 30 21:37:24.014471 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.014445 4926 cpu_manager.go:414] "containerMap: removing container" podUID="2eea6ad7-9d9b-4205-89d2-af24991331de" containerName="istio-proxy" Oct 30 21:37:24.014566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.014473 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eea6ad7-9d9b-4205-89d2-af24991331de" containerName="istio-proxy" Oct 30 21:37:24.014566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.014505 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0" containerName="revoke-keys" Oct 30 21:37:24.014566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.014513 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0" containerName="revoke-keys" Oct 30 21:37:24.014566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.014556 4926 cpu_manager.go:414] "containerMap: removing container" podUID="84b972f9-f7bd-4895-9863-dbf951a09e00" containerName="istio-proxy" Oct 30 21:37:24.014716 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.014566 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b972f9-f7bd-4895-9863-dbf951a09e00" containerName="istio-proxy" Oct 30 21:37:24.014759 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.014746 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0" containerName="revoke-keys" Oct 30 21:37:24.014806 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.014762 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eea6ad7-9d9b-4205-89d2-af24991331de" containerName="istio-proxy" Oct 30 21:37:24.014806 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.014776 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b972f9-f7bd-4895-9863-dbf951a09e00" containerName="istio-proxy" Oct 30 21:37:24.021329 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.021295 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f" Oct 30 21:37:24.038049 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.037989 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f"] Oct 30 21:37:24.041749 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.041716 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:37:24.041981 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.041778 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:37:24.075596 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.075543 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024cdea3-bfac-4a81-8775-6069726d3186-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-79e911-zs45f\" (UID: \"024cdea3-bfac-4a81-8775-6069726d3186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f" Oct 30 21:37:24.075819 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.075604 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024cdea3-bfac-4a81-8775-6069726d3186-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-79e911-zs45f\" (UID: \"024cdea3-bfac-4a81-8775-6069726d3186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f" Oct 30 21:37:24.075819 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.075795 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n25gj\" (UniqueName: \"kubernetes.io/projected/024cdea3-bfac-4a81-8775-6069726d3186-kube-api-access-n25gj\") pod \"maas-api-revoke-keys-e2e-inf-a-79e911-zs45f\" (UID: \"024cdea3-bfac-4a81-8775-6069726d3186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f" Oct 30 21:37:24.177729 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.177685 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024cdea3-bfac-4a81-8775-6069726d3186-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-79e911-zs45f\" (UID: \"024cdea3-bfac-4a81-8775-6069726d3186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f" Oct 30 21:37:24.177977 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.177749 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024cdea3-bfac-4a81-8775-6069726d3186-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-79e911-zs45f\" (UID: \"024cdea3-bfac-4a81-8775-6069726d3186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f" Oct 30 21:37:24.177977 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.177819 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n25gj\" (UniqueName: \"kubernetes.io/projected/024cdea3-bfac-4a81-8775-6069726d3186-kube-api-access-n25gj\") pod \"maas-api-revoke-keys-e2e-inf-a-79e911-zs45f\" (UID: \"024cdea3-bfac-4a81-8775-6069726d3186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f" Oct 30 21:37:24.178419 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.178387 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024cdea3-bfac-4a81-8775-6069726d3186-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-79e911-zs45f\" (UID: \"024cdea3-bfac-4a81-8775-6069726d3186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f" Oct 30 21:37:24.180253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.180227 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024cdea3-bfac-4a81-8775-6069726d3186-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-79e911-zs45f\" (UID: \"024cdea3-bfac-4a81-8775-6069726d3186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f" Oct 30 21:37:24.205563 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.205527 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n25gj\" (UniqueName: \"kubernetes.io/projected/024cdea3-bfac-4a81-8775-6069726d3186-kube-api-access-n25gj\") pod \"maas-api-revoke-keys-e2e-inf-a-79e911-zs45f\" (UID: \"024cdea3-bfac-4a81-8775-6069726d3186\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f" Oct 30 21:37:24.343484 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.343444 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f" Oct 30 21:37:24.661381 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:24.661344 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f"] Oct 30 21:37:24.661534 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:37:24.661420 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024cdea3_bfac_4a81_8775_6069726d3186.slice/crio-4c8d552668ef9ff136c4d4303eb13856aaa6a8bbd1b1b44bdcbbd5b89ffb4b96 WatchSource:0}: Error finding container 4c8d552668ef9ff136c4d4303eb13856aaa6a8bbd1b1b44bdcbbd5b89ffb4b96: Status 404 returned error can't find the container with id 4c8d552668ef9ff136c4d4303eb13856aaa6a8bbd1b1b44bdcbbd5b89ffb4b96 Oct 30 21:37:25.457335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:25.457289 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f" event={"ID":"024cdea3-bfac-4a81-8775-6069726d3186","Type":"ContainerStarted","Data":"c423a11dafa84665c14793b8c51dcc67649f33a31da6d61c55390439f504d06a"} Oct 30 21:37:25.458350 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:25.457363 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f" event={"ID":"024cdea3-bfac-4a81-8775-6069726d3186","Type":"ContainerStarted","Data":"4c8d552668ef9ff136c4d4303eb13856aaa6a8bbd1b1b44bdcbbd5b89ffb4b96"} Oct 30 21:37:25.499792 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:25.499730 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f" podStartSLOduration=2.499714564 podStartE2EDuration="2.499714564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:25.495720863 +0000 UTC m=+2232.616794609" watchObservedRunningTime="2026-10-30 21:37:25.499714564 +0000 UTC m=+2232.620788311" Oct 30 21:37:26.471621 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:26.471584 4926 generic.go:356] "Generic (PLEG): container finished" podID="024cdea3-bfac-4a81-8775-6069726d3186" containerID="c423a11dafa84665c14793b8c51dcc67649f33a31da6d61c55390439f504d06a" exitCode=0 Oct 30 21:37:26.472078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:26.471670 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f" event={"ID":"024cdea3-bfac-4a81-8775-6069726d3186","Type":"ContainerDied","Data":"c423a11dafa84665c14793b8c51dcc67649f33a31da6d61c55390439f504d06a"} Oct 30 21:37:27.658912 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:27.658880 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f" Oct 30 21:37:27.739388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:27.739293 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024cdea3-bfac-4a81-8775-6069726d3186-kube-api-access-n25gj\" (UniqueName: \"kubernetes.io/projected/024cdea3-bfac-4a81-8775-6069726d3186-kube-api-access-n25gj\") pod \"024cdea3-bfac-4a81-8775-6069726d3186\" (UID: \"024cdea3-bfac-4a81-8775-6069726d3186\") " Oct 30 21:37:27.739586 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:27.739440 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/024cdea3-bfac-4a81-8775-6069726d3186-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024cdea3-bfac-4a81-8775-6069726d3186-maas-api-openshift-ca\") pod \"024cdea3-bfac-4a81-8775-6069726d3186\" (UID: \"024cdea3-bfac-4a81-8775-6069726d3186\") " Oct 30 21:37:27.739586 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:27.739491 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/024cdea3-bfac-4a81-8775-6069726d3186-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024cdea3-bfac-4a81-8775-6069726d3186-maas-api-xks-ca\") pod \"024cdea3-bfac-4a81-8775-6069726d3186\" (UID: \"024cdea3-bfac-4a81-8775-6069726d3186\") " Oct 30 21:37:27.739807 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:27.739780 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/024cdea3-bfac-4a81-8775-6069726d3186-maas-api-openshift-ca" pod "024cdea3-bfac-4a81-8775-6069726d3186" (UID: "024cdea3-bfac-4a81-8775-6069726d3186"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:27.739984 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:27.739961 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/024cdea3-bfac-4a81-8775-6069726d3186-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:27.741625 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:27.741598 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024cdea3-bfac-4a81-8775-6069726d3186-kube-api-access-n25gj" pod "024cdea3-bfac-4a81-8775-6069726d3186" (UID: "024cdea3-bfac-4a81-8775-6069726d3186"). InnerVolumeSpecName "kube-api-access-n25gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:27.742069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:27.742042 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024cdea3-bfac-4a81-8775-6069726d3186-maas-api-xks-ca" pod "024cdea3-bfac-4a81-8775-6069726d3186" (UID: "024cdea3-bfac-4a81-8775-6069726d3186"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:27.841641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:27.841605 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/024cdea3-bfac-4a81-8775-6069726d3186-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:27.841641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:27.841638 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n25gj\" (UniqueName: \"kubernetes.io/projected/024cdea3-bfac-4a81-8775-6069726d3186-kube-api-access-n25gj\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:28.493009 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:28.492978 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f" Oct 30 21:37:28.493214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:28.493003 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f" event={"ID":"024cdea3-bfac-4a81-8775-6069726d3186","Type":"ContainerDied","Data":"4c8d552668ef9ff136c4d4303eb13856aaa6a8bbd1b1b44bdcbbd5b89ffb4b96"} Oct 30 21:37:28.493214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:28.493074 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c8d552668ef9ff136c4d4303eb13856aaa6a8bbd1b1b44bdcbbd5b89ffb4b96" Oct 30 21:37:32.032765 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:32.032722 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht"] Oct 30 21:37:32.043855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:32.043820 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6acc995f-2sxht"] Oct 30 21:37:33.020997 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:33.020949 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afb4ba3a-47b7-46b8-a256-9784a904e748" path="/var/lib/kubelet/pods/afb4ba3a-47b7-46b8-a256-9784a904e748/volumes" Oct 30 21:37:33.821024 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:33.820977 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk" podUID="23944deb-c056-466d-b79d-d56a257d98df" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:37:33.821447 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:33.821048 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj" podUID="7f05e57c-842d-4aac-b454-10ca04014727" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:37:34.190699 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:34.190658 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6"] Oct 30 21:37:34.190941 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:34.190912 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" podUID="3e44f1d5-29f6-4129-b244-82dd24a733b8" containerName="maas-api" containerID="cri-o://9a74dd74eaa7130369f442ebcc73fcc5285963eb509e047ea1114e3dfef1681c" gracePeriod=30 Oct 30 21:37:34.552333 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:34.552190 4926 generic.go:356] "Generic (PLEG): container finished" podID="3e44f1d5-29f6-4129-b244-82dd24a733b8" containerID="9a74dd74eaa7130369f442ebcc73fcc5285963eb509e047ea1114e3dfef1681c" exitCode=0 Oct 30 21:37:34.552333 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:34.552309 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" event={"ID":"3e44f1d5-29f6-4129-b244-82dd24a733b8","Type":"ContainerDied","Data":"9a74dd74eaa7130369f442ebcc73fcc5285963eb509e047ea1114e3dfef1681c"} Oct 30 21:37:34.603179 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:34.603135 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" Oct 30 21:37:34.752567 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:34.752527 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e44f1d5-29f6-4129-b244-82dd24a733b8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e44f1d5-29f6-4129-b244-82dd24a733b8-openshift-service-ca\") pod \"3e44f1d5-29f6-4129-b244-82dd24a733b8\" (UID: \"3e44f1d5-29f6-4129-b244-82dd24a733b8\") " Oct 30 21:37:34.754311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:34.754266 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e44f1d5-29f6-4129-b244-82dd24a733b8-kube-api-access-jgnv4\" (UniqueName: \"kubernetes.io/projected/3e44f1d5-29f6-4129-b244-82dd24a733b8-kube-api-access-jgnv4\") pod \"3e44f1d5-29f6-4129-b244-82dd24a733b8\" (UID: \"3e44f1d5-29f6-4129-b244-82dd24a733b8\") " Oct 30 21:37:34.754755 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:34.754729 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e44f1d5-29f6-4129-b244-82dd24a733b8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e44f1d5-29f6-4129-b244-82dd24a733b8-metrics-certs\") pod \"3e44f1d5-29f6-4129-b244-82dd24a733b8\" (UID: \"3e44f1d5-29f6-4129-b244-82dd24a733b8\") " Oct 30 21:37:34.754994 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:34.754974 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e44f1d5-29f6-4129-b244-82dd24a733b8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e44f1d5-29f6-4129-b244-82dd24a733b8-maas-api-tls\") pod \"3e44f1d5-29f6-4129-b244-82dd24a733b8\" (UID: \"3e44f1d5-29f6-4129-b244-82dd24a733b8\") " Oct 30 21:37:34.776064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:34.760506 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e44f1d5-29f6-4129-b244-82dd24a733b8-openshift-service-ca" pod "3e44f1d5-29f6-4129-b244-82dd24a733b8" (UID: "3e44f1d5-29f6-4129-b244-82dd24a733b8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:34.776064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:34.767131 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e44f1d5-29f6-4129-b244-82dd24a733b8-kube-api-access-jgnv4" pod "3e44f1d5-29f6-4129-b244-82dd24a733b8" (UID: "3e44f1d5-29f6-4129-b244-82dd24a733b8"). InnerVolumeSpecName "kube-api-access-jgnv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:34.803186 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:34.802201 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e44f1d5-29f6-4129-b244-82dd24a733b8-maas-api-tls" pod "3e44f1d5-29f6-4129-b244-82dd24a733b8" (UID: "3e44f1d5-29f6-4129-b244-82dd24a733b8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:34.807164 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:34.807127 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e44f1d5-29f6-4129-b244-82dd24a733b8-metrics-certs" pod "3e44f1d5-29f6-4129-b244-82dd24a733b8" (UID: "3e44f1d5-29f6-4129-b244-82dd24a733b8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:34.857073 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:34.856997 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e44f1d5-29f6-4129-b244-82dd24a733b8-maas-api-tls\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:34.857499 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:34.857085 4926 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e44f1d5-29f6-4129-b244-82dd24a733b8-openshift-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:34.857499 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:34.857101 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgnv4\" (UniqueName: \"kubernetes.io/projected/3e44f1d5-29f6-4129-b244-82dd24a733b8-kube-api-access-jgnv4\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:34.857499 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:34.857114 4926 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e44f1d5-29f6-4129-b244-82dd24a733b8-metrics-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:35.564262 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:35.564182 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" Oct 30 21:37:35.564262 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:35.564211 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6" event={"ID":"3e44f1d5-29f6-4129-b244-82dd24a733b8","Type":"ContainerDied","Data":"8debe2ac5c507ac75805b7e93de75e27506e4622e582e9b45f73de8943558c71"} Oct 30 21:37:35.564472 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:35.564281 4926 scope.go:122] "RemoveContainer" containerID="9a74dd74eaa7130369f442ebcc73fcc5285963eb509e047ea1114e3dfef1681c" Oct 30 21:37:35.619309 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:35.619269 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6"] Oct 30 21:37:35.632241 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:35.632206 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-79e911-54988765fb-lkkb6"] Oct 30 21:37:35.737015 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:35.736978 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f"] Oct 30 21:37:35.773934 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:35.773894 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-79e911-zs45f"] Oct 30 21:37:36.544947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:36.544827 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh"] Oct 30 21:37:36.901769 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:36.901675 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h"] Oct 30 21:37:36.902628 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:36.902604 4926 cpu_manager.go:414] "containerMap: removing container" podUID="2eea6ad7-9d9b-4205-89d2-af24991331de" containerName="istio-proxy" Oct 30 21:37:36.902628 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:36.902627 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eea6ad7-9d9b-4205-89d2-af24991331de" containerName="istio-proxy" Oct 30 21:37:36.902803 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:36.902658 4926 cpu_manager.go:414] "containerMap: removing container" podUID="3e44f1d5-29f6-4129-b244-82dd24a733b8" containerName="maas-api" Oct 30 21:37:36.902803 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:36.902666 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e44f1d5-29f6-4129-b244-82dd24a733b8" containerName="maas-api" Oct 30 21:37:36.902803 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:36.902678 4926 cpu_manager.go:414] "containerMap: removing container" podUID="024cdea3-bfac-4a81-8775-6069726d3186" containerName="revoke-keys" Oct 30 21:37:36.902803 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:36.902686 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024cdea3-bfac-4a81-8775-6069726d3186" containerName="revoke-keys" Oct 30 21:37:36.902927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:36.902866 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e44f1d5-29f6-4129-b244-82dd24a733b8" containerName="maas-api" Oct 30 21:37:36.902927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:36.902878 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="024cdea3-bfac-4a81-8775-6069726d3186" containerName="revoke-keys" Oct 30 21:37:36.902927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:36.902892 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eea6ad7-9d9b-4205-89d2-af24991331de" containerName="istio-proxy" Oct 30 21:37:36.908522 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:36.908500 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h" Oct 30 21:37:36.923671 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:36.923645 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h"] Oct 30 21:37:36.998761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:36.998723 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:37:36.998761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:36.998740 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:37:37.017960 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:37.017920 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="024cdea3-bfac-4a81-8775-6069726d3186" path="/var/lib/kubelet/pods/024cdea3-bfac-4a81-8775-6069726d3186/volumes" Oct 30 21:37:37.018536 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:37.018518 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e44f1d5-29f6-4129-b244-82dd24a733b8" path="/var/lib/kubelet/pods/3e44f1d5-29f6-4129-b244-82dd24a733b8/volumes" Oct 30 21:37:37.088061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:37.087838 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlmnr\" (UniqueName: \"kubernetes.io/projected/8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345-kube-api-access-hlmnr\") pod \"maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h\" (UID: \"8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h" Oct 30 21:37:37.088061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:37.087944 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h\" (UID: \"8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h" Oct 30 21:37:37.088061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:37.087976 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h\" (UID: \"8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h" Oct 30 21:37:37.189249 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:37.189205 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlmnr\" (UniqueName: \"kubernetes.io/projected/8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345-kube-api-access-hlmnr\") pod \"maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h\" (UID: \"8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h" Oct 30 21:37:37.189465 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:37.189308 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h\" (UID: \"8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h" Oct 30 21:37:37.189465 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:37.189344 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h\" (UID: \"8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h" Oct 30 21:37:37.190010 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:37.189979 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h\" (UID: \"8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h" Oct 30 21:37:37.192130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:37.192104 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h\" (UID: \"8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h" Oct 30 21:37:37.208884 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:37.208849 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlmnr\" (UniqueName: \"kubernetes.io/projected/8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345-kube-api-access-hlmnr\") pod \"maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h\" (UID: \"8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h" Oct 30 21:37:37.231542 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:37.231506 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h" Oct 30 21:37:37.491122 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:37:37.489925 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:37:37.558457 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:37.558425 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h"] Oct 30 21:37:37.560165 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:37:37.560139 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6b1d8f_f9cf_41ee_bbe4_8b4fc9a7e345.slice/crio-7d48e5b03357b7aebf4672b45169da322353027b2b936add4099ed0bebe4f74e WatchSource:0}: Error finding container 7d48e5b03357b7aebf4672b45169da322353027b2b936add4099ed0bebe4f74e: Status 404 returned error can't find the container with id 7d48e5b03357b7aebf4672b45169da322353027b2b936add4099ed0bebe4f74e Oct 30 21:37:37.597497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:37.597453 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h" event={"ID":"8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345","Type":"ContainerStarted","Data":"7d48e5b03357b7aebf4672b45169da322353027b2b936add4099ed0bebe4f74e"} Oct 30 21:37:38.611469 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:38.611430 4926 generic.go:356] "Generic (PLEG): container finished" podID="8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345" containerID="507e4109b3ff2893141ad527d690b3ff8e98672eb614a74be6ce481c5f1b9269" exitCode=0 Oct 30 21:37:38.611990 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:38.611515 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h" event={"ID":"8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345","Type":"ContainerDied","Data":"507e4109b3ff2893141ad527d690b3ff8e98672eb614a74be6ce481c5f1b9269"} Oct 30 21:37:39.790535 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:39.790499 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h" Oct 30 21:37:39.924656 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:39.924620 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345-kube-api-access-hlmnr\" (UniqueName: \"kubernetes.io/projected/8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345-kube-api-access-hlmnr\") pod \"8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345\" (UID: \"8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345\") " Oct 30 21:37:39.924821 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:39.924713 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345-maas-api-openshift-ca\") pod \"8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345\" (UID: \"8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345\") " Oct 30 21:37:39.924821 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:39.924799 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345-maas-api-xks-ca\") pod \"8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345\" (UID: \"8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345\") " Oct 30 21:37:39.925133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:39.925103 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345-maas-api-openshift-ca" pod "8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345" (UID: "8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:39.925290 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:39.925269 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:39.927099 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:39.927069 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345-kube-api-access-hlmnr" pod "8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345" (UID: "8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345"). InnerVolumeSpecName "kube-api-access-hlmnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:39.927201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:39.927105 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345-maas-api-xks-ca" pod "8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345" (UID: "8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:40.026308 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:40.026267 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:40.026308 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:40.026301 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlmnr\" (UniqueName: \"kubernetes.io/projected/8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345-kube-api-access-hlmnr\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:40.635235 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:40.635194 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h" Oct 30 21:37:40.635235 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:40.635217 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h" event={"ID":"8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345","Type":"ContainerDied","Data":"7d48e5b03357b7aebf4672b45169da322353027b2b936add4099ed0bebe4f74e"} Oct 30 21:37:40.635479 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:40.635280 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d48e5b03357b7aebf4672b45169da322353027b2b936add4099ed0bebe4f74e" Oct 30 21:37:43.176255 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.176216 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj" Oct 30 21:37:43.180246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.180216 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk" Oct 30 21:37:43.219280 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.219238 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv"] Oct 30 21:37:43.219868 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.219848 4926 cpu_manager.go:414] "containerMap: removing container" podUID="7f05e57c-842d-4aac-b454-10ca04014727" containerName="payload-processing" Oct 30 21:37:43.219868 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.219866 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f05e57c-842d-4aac-b454-10ca04014727" containerName="payload-processing" Oct 30 21:37:43.219993 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.219884 4926 cpu_manager.go:414] "containerMap: removing container" podUID="8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345" containerName="revoke-keys" Oct 30 21:37:43.219993 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.219890 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345" containerName="revoke-keys" Oct 30 21:37:43.219993 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.219910 4926 cpu_manager.go:414] "containerMap: removing container" podUID="23944deb-c056-466d-b79d-d56a257d98df" containerName="payload-pre-processing" Oct 30 21:37:43.219993 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.219917 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23944deb-c056-466d-b79d-d56a257d98df" containerName="payload-pre-processing" Oct 30 21:37:43.220196 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.220025 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="23944deb-c056-466d-b79d-d56a257d98df" containerName="payload-pre-processing" Oct 30 21:37:43.220196 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.220053 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f05e57c-842d-4aac-b454-10ca04014727" containerName="payload-processing" Oct 30 21:37:43.220196 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.220063 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345" containerName="revoke-keys" Oct 30 21:37:43.223917 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.223889 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv" Oct 30 21:37:43.242913 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.242874 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv"] Oct 30 21:37:43.285302 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.285250 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23944deb-c056-466d-b79d-d56a257d98df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23944deb-c056-466d-b79d-d56a257d98df-plugins-config-volume\") pod \"23944deb-c056-466d-b79d-d56a257d98df\" (UID: \"23944deb-c056-466d-b79d-d56a257d98df\") " Oct 30 21:37:43.285508 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.285357 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23944deb-c056-466d-b79d-d56a257d98df-kube-api-access-45shf\" (UniqueName: \"kubernetes.io/projected/23944deb-c056-466d-b79d-d56a257d98df-kube-api-access-45shf\") pod \"23944deb-c056-466d-b79d-d56a257d98df\" (UID: \"23944deb-c056-466d-b79d-d56a257d98df\") " Oct 30 21:37:43.285569 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.285512 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f05e57c-842d-4aac-b454-10ca04014727-kube-api-access-xdwxn\" (UniqueName: \"kubernetes.io/projected/7f05e57c-842d-4aac-b454-10ca04014727-kube-api-access-xdwxn\") pod \"7f05e57c-842d-4aac-b454-10ca04014727\" (UID: \"7f05e57c-842d-4aac-b454-10ca04014727\") " Oct 30 21:37:43.285569 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.285563 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f05e57c-842d-4aac-b454-10ca04014727-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f05e57c-842d-4aac-b454-10ca04014727-plugins-config-volume\") pod \"7f05e57c-842d-4aac-b454-10ca04014727\" (UID: \"7f05e57c-842d-4aac-b454-10ca04014727\") " Oct 30 21:37:43.286231 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.286184 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f05e57c-842d-4aac-b454-10ca04014727-plugins-config-volume" pod "7f05e57c-842d-4aac-b454-10ca04014727" (UID: "7f05e57c-842d-4aac-b454-10ca04014727"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:43.286365 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.286246 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23944deb-c056-466d-b79d-d56a257d98df-plugins-config-volume" pod "23944deb-c056-466d-b79d-d56a257d98df" (UID: "23944deb-c056-466d-b79d-d56a257d98df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:43.286495 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.286469 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059be9e3-5689-4c00-9554-f7c9f6a2cf59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv\" (UID: \"059be9e3-5689-4c00-9554-f7c9f6a2cf59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv" Oct 30 21:37:43.286714 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.286691 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059be9e3-5689-4c00-9554-f7c9f6a2cf59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv\" (UID: \"059be9e3-5689-4c00-9554-f7c9f6a2cf59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv" Oct 30 21:37:43.287118 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.287098 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w5hq\" (UniqueName: \"kubernetes.io/projected/059be9e3-5689-4c00-9554-f7c9f6a2cf59-kube-api-access-5w5hq\") pod \"maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv\" (UID: \"059be9e3-5689-4c00-9554-f7c9f6a2cf59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv" Oct 30 21:37:43.287443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.287423 4926 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f05e57c-842d-4aac-b454-10ca04014727-plugins-config-volume\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:43.287549 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.287536 4926 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23944deb-c056-466d-b79d-d56a257d98df-plugins-config-volume\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:43.288951 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.288799 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f05e57c-842d-4aac-b454-10ca04014727-kube-api-access-xdwxn" pod "7f05e57c-842d-4aac-b454-10ca04014727" (UID: "7f05e57c-842d-4aac-b454-10ca04014727"). InnerVolumeSpecName "kube-api-access-xdwxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:43.290240 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.290214 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23944deb-c056-466d-b79d-d56a257d98df-kube-api-access-45shf" pod "23944deb-c056-466d-b79d-d56a257d98df" (UID: "23944deb-c056-466d-b79d-d56a257d98df"). InnerVolumeSpecName "kube-api-access-45shf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:43.312222 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.312181 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:37:43.312393 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.312228 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:37:43.389388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.389296 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059be9e3-5689-4c00-9554-f7c9f6a2cf59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv\" (UID: \"059be9e3-5689-4c00-9554-f7c9f6a2cf59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv" Oct 30 21:37:43.389388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.389367 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059be9e3-5689-4c00-9554-f7c9f6a2cf59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv\" (UID: \"059be9e3-5689-4c00-9554-f7c9f6a2cf59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv" Oct 30 21:37:43.389647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.389443 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w5hq\" (UniqueName: \"kubernetes.io/projected/059be9e3-5689-4c00-9554-f7c9f6a2cf59-kube-api-access-5w5hq\") pod \"maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv\" (UID: \"059be9e3-5689-4c00-9554-f7c9f6a2cf59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv" Oct 30 21:37:43.389647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.389533 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45shf\" (UniqueName: \"kubernetes.io/projected/23944deb-c056-466d-b79d-d56a257d98df-kube-api-access-45shf\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:43.389647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.389552 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdwxn\" (UniqueName: \"kubernetes.io/projected/7f05e57c-842d-4aac-b454-10ca04014727-kube-api-access-xdwxn\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:43.390296 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.390262 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059be9e3-5689-4c00-9554-f7c9f6a2cf59-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv\" (UID: \"059be9e3-5689-4c00-9554-f7c9f6a2cf59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv" Oct 30 21:37:43.391924 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.391900 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059be9e3-5689-4c00-9554-f7c9f6a2cf59-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv\" (UID: \"059be9e3-5689-4c00-9554-f7c9f6a2cf59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv" Oct 30 21:37:43.412777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.412747 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w5hq\" (UniqueName: \"kubernetes.io/projected/059be9e3-5689-4c00-9554-f7c9f6a2cf59-kube-api-access-5w5hq\") pod \"maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv\" (UID: \"059be9e3-5689-4c00-9554-f7c9f6a2cf59\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv" Oct 30 21:37:43.544938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.544905 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv" Oct 30 21:37:43.674084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.674014 4926 generic.go:356] "Generic (PLEG): container finished" podID="23944deb-c056-466d-b79d-d56a257d98df" containerID="2e2c63096b7b3ea96867edb30086c08735468c0ff58091580e06ac1f49b57bee" exitCode=0 Oct 30 21:37:43.674272 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.674247 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk" event={"ID":"23944deb-c056-466d-b79d-d56a257d98df","Type":"ContainerDied","Data":"2e2c63096b7b3ea96867edb30086c08735468c0ff58091580e06ac1f49b57bee"} Oct 30 21:37:43.674331 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.674314 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk" event={"ID":"23944deb-c056-466d-b79d-d56a257d98df","Type":"ContainerDied","Data":"0bddcf24f1e2fb1dbf7cd68539cb1fa7728a27b4241a6a02939bd63ee7699c64"} Oct 30 21:37:43.674368 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.674352 4926 scope.go:122] "RemoveContainer" containerID="2e2c63096b7b3ea96867edb30086c08735468c0ff58091580e06ac1f49b57bee" Oct 30 21:37:43.674539 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.674524 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk" Oct 30 21:37:43.680644 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.680605 4926 generic.go:356] "Generic (PLEG): container finished" podID="7f05e57c-842d-4aac-b454-10ca04014727" containerID="a19cb5d2133186dd977067429408f91404bb6231735c6dc3b243ec0cf3fa4b93" exitCode=0 Oct 30 21:37:43.680785 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.680750 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj" event={"ID":"7f05e57c-842d-4aac-b454-10ca04014727","Type":"ContainerDied","Data":"a19cb5d2133186dd977067429408f91404bb6231735c6dc3b243ec0cf3fa4b93"} Oct 30 21:37:43.680869 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.680851 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj" event={"ID":"7f05e57c-842d-4aac-b454-10ca04014727","Type":"ContainerDied","Data":"e9800f40afb497d7e534d77f244eccd9eb82cba46cbab5b3745ad5f47729b913"} Oct 30 21:37:43.680986 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.680970 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj" Oct 30 21:37:43.697891 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.697790 4926 scope.go:122] "RemoveContainer" containerID="2e2c63096b7b3ea96867edb30086c08735468c0ff58091580e06ac1f49b57bee" Oct 30 21:37:43.698840 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:37:43.698692 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2c63096b7b3ea96867edb30086c08735468c0ff58091580e06ac1f49b57bee\": container with ID starting with 2e2c63096b7b3ea96867edb30086c08735468c0ff58091580e06ac1f49b57bee not found: ID does not exist" containerID="2e2c63096b7b3ea96867edb30086c08735468c0ff58091580e06ac1f49b57bee" Oct 30 21:37:43.698840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.698731 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2c63096b7b3ea96867edb30086c08735468c0ff58091580e06ac1f49b57bee"} err="failed to get container status \"2e2c63096b7b3ea96867edb30086c08735468c0ff58091580e06ac1f49b57bee\": rpc error: code = NotFound desc = could not find container \"2e2c63096b7b3ea96867edb30086c08735468c0ff58091580e06ac1f49b57bee\": container with ID starting with 2e2c63096b7b3ea96867edb30086c08735468c0ff58091580e06ac1f49b57bee not found: ID does not exist" Oct 30 21:37:43.698840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.698754 4926 scope.go:122] "RemoveContainer" containerID="a19cb5d2133186dd977067429408f91404bb6231735c6dc3b243ec0cf3fa4b93" Oct 30 21:37:43.716220 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.716083 4926 scope.go:122] "RemoveContainer" containerID="a19cb5d2133186dd977067429408f91404bb6231735c6dc3b243ec0cf3fa4b93" Oct 30 21:37:43.716489 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:37:43.716462 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a19cb5d2133186dd977067429408f91404bb6231735c6dc3b243ec0cf3fa4b93\": container with ID starting with a19cb5d2133186dd977067429408f91404bb6231735c6dc3b243ec0cf3fa4b93 not found: ID does not exist" containerID="a19cb5d2133186dd977067429408f91404bb6231735c6dc3b243ec0cf3fa4b93" Oct 30 21:37:43.716555 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.716501 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a19cb5d2133186dd977067429408f91404bb6231735c6dc3b243ec0cf3fa4b93"} err="failed to get container status \"a19cb5d2133186dd977067429408f91404bb6231735c6dc3b243ec0cf3fa4b93\": rpc error: code = NotFound desc = could not find container \"a19cb5d2133186dd977067429408f91404bb6231735c6dc3b243ec0cf3fa4b93\": container with ID starting with a19cb5d2133186dd977067429408f91404bb6231735c6dc3b243ec0cf3fa4b93 not found: ID does not exist" Oct 30 21:37:43.770692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.770635 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj"] Oct 30 21:37:43.775170 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.775138 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-200d11-5f9d5865f8-qx6rj"] Oct 30 21:37:43.824946 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.824906 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk"] Oct 30 21:37:43.834087 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.834056 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-200d11-99fbc69b7-b54sk"] Oct 30 21:37:43.877514 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:43.877478 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv"] Oct 30 21:37:43.878353 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:37:43.878322 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod059be9e3_5689_4c00_9554_f7c9f6a2cf59.slice/crio-5ea8d2f4d81c3f167585d13c578fad61dc0d273bd3e2447baf700dd0a78b4722 WatchSource:0}: Error finding container 5ea8d2f4d81c3f167585d13c578fad61dc0d273bd3e2447baf700dd0a78b4722: Status 404 returned error can't find the container with id 5ea8d2f4d81c3f167585d13c578fad61dc0d273bd3e2447baf700dd0a78b4722 Oct 30 21:37:44.699819 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:44.699774 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv" event={"ID":"059be9e3-5689-4c00-9554-f7c9f6a2cf59","Type":"ContainerStarted","Data":"31d29ef57e62151ad655b0cc348085d1421ecf8b2de7adb70a584d14a7721b2d"} Oct 30 21:37:44.700357 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:44.699843 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv" event={"ID":"059be9e3-5689-4c00-9554-f7c9f6a2cf59","Type":"ContainerStarted","Data":"5ea8d2f4d81c3f167585d13c578fad61dc0d273bd3e2447baf700dd0a78b4722"} Oct 30 21:37:44.738331 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:44.738272 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv" podStartSLOduration=1.7382584479999998 podStartE2EDuration="1.738258448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:44.737550414 +0000 UTC m=+2251.858624162" watchObservedRunningTime="2026-10-30 21:37:44.738258448 +0000 UTC m=+2251.859332194" Oct 30 21:37:45.020363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:45.020281 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23944deb-c056-466d-b79d-d56a257d98df" path="/var/lib/kubelet/pods/23944deb-c056-466d-b79d-d56a257d98df/volumes" Oct 30 21:37:45.021188 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:45.021147 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f05e57c-842d-4aac-b454-10ca04014727" path="/var/lib/kubelet/pods/7f05e57c-842d-4aac-b454-10ca04014727/volumes" Oct 30 21:37:45.498390 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:45.498344 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:37:45.715782 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:45.715739 4926 generic.go:356] "Generic (PLEG): container finished" podID="059be9e3-5689-4c00-9554-f7c9f6a2cf59" containerID="31d29ef57e62151ad655b0cc348085d1421ecf8b2de7adb70a584d14a7721b2d" exitCode=0 Oct 30 21:37:45.716376 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:45.715792 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv" event={"ID":"059be9e3-5689-4c00-9554-f7c9f6a2cf59","Type":"ContainerDied","Data":"31d29ef57e62151ad655b0cc348085d1421ecf8b2de7adb70a584d14a7721b2d"} Oct 30 21:37:47.037150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:47.037011 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv" Oct 30 21:37:47.061663 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:47.061628 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/059be9e3-5689-4c00-9554-f7c9f6a2cf59-kube-api-access-5w5hq\" (UniqueName: \"kubernetes.io/projected/059be9e3-5689-4c00-9554-f7c9f6a2cf59-kube-api-access-5w5hq\") pod \"059be9e3-5689-4c00-9554-f7c9f6a2cf59\" (UID: \"059be9e3-5689-4c00-9554-f7c9f6a2cf59\") " Oct 30 21:37:47.061855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:47.061733 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/059be9e3-5689-4c00-9554-f7c9f6a2cf59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059be9e3-5689-4c00-9554-f7c9f6a2cf59-maas-api-xks-ca\") pod \"059be9e3-5689-4c00-9554-f7c9f6a2cf59\" (UID: \"059be9e3-5689-4c00-9554-f7c9f6a2cf59\") " Oct 30 21:37:47.061855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:47.061780 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/059be9e3-5689-4c00-9554-f7c9f6a2cf59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059be9e3-5689-4c00-9554-f7c9f6a2cf59-maas-api-openshift-ca\") pod \"059be9e3-5689-4c00-9554-f7c9f6a2cf59\" (UID: \"059be9e3-5689-4c00-9554-f7c9f6a2cf59\") " Oct 30 21:37:47.062199 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:47.062172 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/059be9e3-5689-4c00-9554-f7c9f6a2cf59-maas-api-openshift-ca" pod "059be9e3-5689-4c00-9554-f7c9f6a2cf59" (UID: "059be9e3-5689-4c00-9554-f7c9f6a2cf59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:47.062637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:47.062608 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/059be9e3-5689-4c00-9554-f7c9f6a2cf59-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:47.064424 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:47.064402 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/059be9e3-5689-4c00-9554-f7c9f6a2cf59-kube-api-access-5w5hq" pod "059be9e3-5689-4c00-9554-f7c9f6a2cf59" (UID: "059be9e3-5689-4c00-9554-f7c9f6a2cf59"). InnerVolumeSpecName "kube-api-access-5w5hq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:47.064506 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:47.064400 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/059be9e3-5689-4c00-9554-f7c9f6a2cf59-maas-api-xks-ca" pod "059be9e3-5689-4c00-9554-f7c9f6a2cf59" (UID: "059be9e3-5689-4c00-9554-f7c9f6a2cf59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:47.163709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:47.163668 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w5hq\" (UniqueName: \"kubernetes.io/projected/059be9e3-5689-4c00-9554-f7c9f6a2cf59-kube-api-access-5w5hq\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:47.163709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:47.163704 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/059be9e3-5689-4c00-9554-f7c9f6a2cf59-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:47.792850 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:47.792816 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv" Oct 30 21:37:47.793082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:47.792847 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv" event={"ID":"059be9e3-5689-4c00-9554-f7c9f6a2cf59","Type":"ContainerDied","Data":"5ea8d2f4d81c3f167585d13c578fad61dc0d273bd3e2447baf700dd0a78b4722"} Oct 30 21:37:47.793082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:47.792897 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ea8d2f4d81c3f167585d13c578fad61dc0d273bd3e2447baf700dd0a78b4722" Oct 30 21:37:48.991264 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:48.991223 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b"] Oct 30 21:37:48.991752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:48.991469 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" podUID="be8482db-1f53-4f48-bf7a-fd17a081c6f9" containerName="maas-api" containerID="cri-o://c6e690d2ef345bc09cec63bd149db7b5ee5ce7a85af07b25df3aa05183b6b653" gracePeriod=30 Oct 30 21:37:49.360219 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.360188 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" Oct 30 21:37:49.459127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.458501 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be8482db-1f53-4f48-bf7a-fd17a081c6f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8482db-1f53-4f48-bf7a-fd17a081c6f9-openshift-service-ca\") pod \"be8482db-1f53-4f48-bf7a-fd17a081c6f9\" (UID: \"be8482db-1f53-4f48-bf7a-fd17a081c6f9\") " Oct 30 21:37:49.459127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.458608 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be8482db-1f53-4f48-bf7a-fd17a081c6f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be8482db-1f53-4f48-bf7a-fd17a081c6f9-maas-api-tls\") pod \"be8482db-1f53-4f48-bf7a-fd17a081c6f9\" (UID: \"be8482db-1f53-4f48-bf7a-fd17a081c6f9\") " Oct 30 21:37:49.459127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.458685 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be8482db-1f53-4f48-bf7a-fd17a081c6f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/be8482db-1f53-4f48-bf7a-fd17a081c6f9-metrics-certs\") pod \"be8482db-1f53-4f48-bf7a-fd17a081c6f9\" (UID: \"be8482db-1f53-4f48-bf7a-fd17a081c6f9\") " Oct 30 21:37:49.459127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.458758 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be8482db-1f53-4f48-bf7a-fd17a081c6f9-kube-api-access-k9cvd\" (UniqueName: \"kubernetes.io/projected/be8482db-1f53-4f48-bf7a-fd17a081c6f9-kube-api-access-k9cvd\") pod \"be8482db-1f53-4f48-bf7a-fd17a081c6f9\" (UID: \"be8482db-1f53-4f48-bf7a-fd17a081c6f9\") " Oct 30 21:37:49.461219 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.461186 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be8482db-1f53-4f48-bf7a-fd17a081c6f9-openshift-service-ca" pod "be8482db-1f53-4f48-bf7a-fd17a081c6f9" (UID: "be8482db-1f53-4f48-bf7a-fd17a081c6f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:49.476689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.476606 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be8482db-1f53-4f48-bf7a-fd17a081c6f9-kube-api-access-k9cvd" pod "be8482db-1f53-4f48-bf7a-fd17a081c6f9" (UID: "be8482db-1f53-4f48-bf7a-fd17a081c6f9"). InnerVolumeSpecName "kube-api-access-k9cvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:49.477343 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.477301 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8482db-1f53-4f48-bf7a-fd17a081c6f9-maas-api-tls" pod "be8482db-1f53-4f48-bf7a-fd17a081c6f9" (UID: "be8482db-1f53-4f48-bf7a-fd17a081c6f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:49.478020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.477987 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be8482db-1f53-4f48-bf7a-fd17a081c6f9-metrics-certs" pod "be8482db-1f53-4f48-bf7a-fd17a081c6f9" (UID: "be8482db-1f53-4f48-bf7a-fd17a081c6f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:49.561903 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.561793 4926 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/be8482db-1f53-4f48-bf7a-fd17a081c6f9-metrics-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:49.561903 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.561833 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9cvd\" (UniqueName: \"kubernetes.io/projected/be8482db-1f53-4f48-bf7a-fd17a081c6f9-kube-api-access-k9cvd\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:49.561903 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.561849 4926 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/be8482db-1f53-4f48-bf7a-fd17a081c6f9-openshift-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:49.561903 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.561864 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/be8482db-1f53-4f48-bf7a-fd17a081c6f9-maas-api-tls\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:49.777995 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.777947 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm"] Oct 30 21:37:49.782924 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.782889 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-200d11-sk5wm"] Oct 30 21:37:49.811102 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.811045 4926 generic.go:356] "Generic (PLEG): container finished" podID="be8482db-1f53-4f48-bf7a-fd17a081c6f9" containerID="c6e690d2ef345bc09cec63bd149db7b5ee5ce7a85af07b25df3aa05183b6b653" exitCode=0 Oct 30 21:37:49.811102 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.811089 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" Oct 30 21:37:49.811102 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.811095 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" event={"ID":"be8482db-1f53-4f48-bf7a-fd17a081c6f9","Type":"ContainerDied","Data":"c6e690d2ef345bc09cec63bd149db7b5ee5ce7a85af07b25df3aa05183b6b653"} Oct 30 21:37:49.811452 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.811162 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b" event={"ID":"be8482db-1f53-4f48-bf7a-fd17a081c6f9","Type":"ContainerDied","Data":"a75a4d41f7a19cc2dec6be6d27b41dc889f849ed7f2c64de66e88e2672e6753e"} Oct 30 21:37:49.811452 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.811191 4926 scope.go:122] "RemoveContainer" containerID="c6e690d2ef345bc09cec63bd149db7b5ee5ce7a85af07b25df3aa05183b6b653" Oct 30 21:37:49.829247 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.829213 4926 scope.go:122] "RemoveContainer" containerID="c6e690d2ef345bc09cec63bd149db7b5ee5ce7a85af07b25df3aa05183b6b653" Oct 30 21:37:49.829624 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:37:49.829599 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6e690d2ef345bc09cec63bd149db7b5ee5ce7a85af07b25df3aa05183b6b653\": container with ID starting with c6e690d2ef345bc09cec63bd149db7b5ee5ce7a85af07b25df3aa05183b6b653 not found: ID does not exist" containerID="c6e690d2ef345bc09cec63bd149db7b5ee5ce7a85af07b25df3aa05183b6b653" Oct 30 21:37:49.829730 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.829636 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6e690d2ef345bc09cec63bd149db7b5ee5ce7a85af07b25df3aa05183b6b653"} err="failed to get container status \"c6e690d2ef345bc09cec63bd149db7b5ee5ce7a85af07b25df3aa05183b6b653\": rpc error: code = NotFound desc = could not find container \"c6e690d2ef345bc09cec63bd149db7b5ee5ce7a85af07b25df3aa05183b6b653\": container with ID starting with c6e690d2ef345bc09cec63bd149db7b5ee5ce7a85af07b25df3aa05183b6b653 not found: ID does not exist" Oct 30 21:37:49.862879 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.862828 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b"] Oct 30 21:37:49.878668 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:49.871345 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-200d11-78ccb66479-6dz2b"] Oct 30 21:37:51.018835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.018788 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="051a4c83-ae7a-47e1-988d-70724bc221ec" path="/var/lib/kubelet/pods/051a4c83-ae7a-47e1-988d-70724bc221ec/volumes" Oct 30 21:37:51.019477 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.019450 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be8482db-1f53-4f48-bf7a-fd17a081c6f9" path="/var/lib/kubelet/pods/be8482db-1f53-4f48-bf7a-fd17a081c6f9/volumes" Oct 30 21:37:51.221190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.221151 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w"] Oct 30 21:37:51.221443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.221411 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" podUID="87506f32-cd27-497b-be39-ef8ae28faef0" containerName="maas-api" containerID="cri-o://80d2a56210390c907286427da1ea74cd2c4dcf8b16af5bb3efac2780dbe57d45" gracePeriod=30 Oct 30 21:37:51.599803 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.597236 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" podUID="aac5cd64-6203-43ee-aa05-d5b8c1e0240a" containerName="main" containerID="cri-o://5e201a253f5363bfedf22643a9fed3a1fe43e5a4458d9036928bea8ea4b902a8" gracePeriod=45 Oct 30 21:37:51.776608 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.776570 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" Oct 30 21:37:51.845968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.836199 4926 generic.go:356] "Generic (PLEG): container finished" podID="aac5cd64-6203-43ee-aa05-d5b8c1e0240a" containerID="5e201a253f5363bfedf22643a9fed3a1fe43e5a4458d9036928bea8ea4b902a8" exitCode=0 Oct 30 21:37:51.845968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.836272 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" event={"ID":"aac5cd64-6203-43ee-aa05-d5b8c1e0240a","Type":"ContainerDied","Data":"5e201a253f5363bfedf22643a9fed3a1fe43e5a4458d9036928bea8ea4b902a8"} Oct 30 21:37:51.845968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.837828 4926 generic.go:356] "Generic (PLEG): container finished" podID="87506f32-cd27-497b-be39-ef8ae28faef0" containerID="80d2a56210390c907286427da1ea74cd2c4dcf8b16af5bb3efac2780dbe57d45" exitCode=0 Oct 30 21:37:51.845968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.838133 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" event={"ID":"87506f32-cd27-497b-be39-ef8ae28faef0","Type":"ContainerDied","Data":"80d2a56210390c907286427da1ea74cd2c4dcf8b16af5bb3efac2780dbe57d45"} Oct 30 21:37:51.845968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.838158 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" Oct 30 21:37:51.845968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.838184 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w" event={"ID":"87506f32-cd27-497b-be39-ef8ae28faef0","Type":"ContainerDied","Data":"df65641218f1b1f4ad01409d589f2854d770278f240c811e2fec87ff98c242a1"} Oct 30 21:37:51.845968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.838214 4926 scope.go:122] "RemoveContainer" containerID="80d2a56210390c907286427da1ea74cd2c4dcf8b16af5bb3efac2780dbe57d45" Oct 30 21:37:51.854949 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.854510 4926 scope.go:122] "RemoveContainer" containerID="80d2a56210390c907286427da1ea74cd2c4dcf8b16af5bb3efac2780dbe57d45" Oct 30 21:37:51.854949 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:37:51.854861 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80d2a56210390c907286427da1ea74cd2c4dcf8b16af5bb3efac2780dbe57d45\": container with ID starting with 80d2a56210390c907286427da1ea74cd2c4dcf8b16af5bb3efac2780dbe57d45 not found: ID does not exist" containerID="80d2a56210390c907286427da1ea74cd2c4dcf8b16af5bb3efac2780dbe57d45" Oct 30 21:37:51.854949 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.854901 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80d2a56210390c907286427da1ea74cd2c4dcf8b16af5bb3efac2780dbe57d45"} err="failed to get container status \"80d2a56210390c907286427da1ea74cd2c4dcf8b16af5bb3efac2780dbe57d45\": rpc error: code = NotFound desc = could not find container \"80d2a56210390c907286427da1ea74cd2c4dcf8b16af5bb3efac2780dbe57d45\": container with ID starting with 80d2a56210390c907286427da1ea74cd2c4dcf8b16af5bb3efac2780dbe57d45 not found: ID does not exist" Oct 30 21:37:51.918075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.917430 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87506f32-cd27-497b-be39-ef8ae28faef0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87506f32-cd27-497b-be39-ef8ae28faef0-maas-api-tls\") pod \"87506f32-cd27-497b-be39-ef8ae28faef0\" (UID: \"87506f32-cd27-497b-be39-ef8ae28faef0\") " Oct 30 21:37:51.918075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.917506 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87506f32-cd27-497b-be39-ef8ae28faef0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87506f32-cd27-497b-be39-ef8ae28faef0-openshift-service-ca\") pod \"87506f32-cd27-497b-be39-ef8ae28faef0\" (UID: \"87506f32-cd27-497b-be39-ef8ae28faef0\") " Oct 30 21:37:51.918075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.917596 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87506f32-cd27-497b-be39-ef8ae28faef0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/87506f32-cd27-497b-be39-ef8ae28faef0-metrics-certs\") pod \"87506f32-cd27-497b-be39-ef8ae28faef0\" (UID: \"87506f32-cd27-497b-be39-ef8ae28faef0\") " Oct 30 21:37:51.918075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.917625 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87506f32-cd27-497b-be39-ef8ae28faef0-kube-api-access-9l69h\" (UniqueName: \"kubernetes.io/projected/87506f32-cd27-497b-be39-ef8ae28faef0-kube-api-access-9l69h\") pod \"87506f32-cd27-497b-be39-ef8ae28faef0\" (UID: \"87506f32-cd27-497b-be39-ef8ae28faef0\") " Oct 30 21:37:51.918859 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.918828 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87506f32-cd27-497b-be39-ef8ae28faef0-openshift-service-ca" pod "87506f32-cd27-497b-be39-ef8ae28faef0" (UID: "87506f32-cd27-497b-be39-ef8ae28faef0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:37:51.921165 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.921132 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87506f32-cd27-497b-be39-ef8ae28faef0-kube-api-access-9l69h" pod "87506f32-cd27-497b-be39-ef8ae28faef0" (UID: "87506f32-cd27-497b-be39-ef8ae28faef0"). InnerVolumeSpecName "kube-api-access-9l69h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:51.922467 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.922425 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87506f32-cd27-497b-be39-ef8ae28faef0-metrics-certs" pod "87506f32-cd27-497b-be39-ef8ae28faef0" (UID: "87506f32-cd27-497b-be39-ef8ae28faef0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:51.924176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.924150 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87506f32-cd27-497b-be39-ef8ae28faef0-maas-api-tls" pod "87506f32-cd27-497b-be39-ef8ae28faef0" (UID: "87506f32-cd27-497b-be39-ef8ae28faef0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:51.999900 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:51.999869 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:37:52.019290 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.019253 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/87506f32-cd27-497b-be39-ef8ae28faef0-maas-api-tls\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:52.019290 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.019288 4926 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/87506f32-cd27-497b-be39-ef8ae28faef0-openshift-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:52.019757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.019302 4926 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87506f32-cd27-497b-be39-ef8ae28faef0-metrics-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:52.019757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.019315 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l69h\" (UniqueName: \"kubernetes.io/projected/87506f32-cd27-497b-be39-ef8ae28faef0-kube-api-access-9l69h\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:52.028961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.028913 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c76951-s96k6"] Oct 30 21:37:52.033530 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.033461 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44c76951-s96k6"] Oct 30 21:37:52.038916 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.038469 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3b28bc-79422"] Oct 30 21:37:52.044236 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.044210 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3b28bc-79422"] Oct 30 21:37:52.121334 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.121227 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-tmp-dir\") pod \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " Oct 30 21:37:52.121334 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.121285 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-kube-api-access-87zd8\" (UniqueName: \"kubernetes.io/projected/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-kube-api-access-87zd8\") pod \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " Oct 30 21:37:52.121334 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.121314 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-model-cache\") pod \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " Oct 30 21:37:52.121645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.121345 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-home\" (UniqueName: \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-home\") pod \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " Oct 30 21:37:52.121645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.121436 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-tls-certs\" (UniqueName: \"kubernetes.io/secret/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-tls-certs\") pod \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " Oct 30 21:37:52.121645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.121489 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-dshm\") pod \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\" (UID: \"aac5cd64-6203-43ee-aa05-d5b8c1e0240a\") " Oct 30 21:37:52.121813 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.121745 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-tmp-dir" pod "aac5cd64-6203-43ee-aa05-d5b8c1e0240a" (UID: "aac5cd64-6203-43ee-aa05-d5b8c1e0240a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:52.122078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.122056 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:52.122307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.122283 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-model-cache" pod "aac5cd64-6203-43ee-aa05-d5b8c1e0240a" (UID: "aac5cd64-6203-43ee-aa05-d5b8c1e0240a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:52.122549 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.122526 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-home" pod "aac5cd64-6203-43ee-aa05-d5b8c1e0240a" (UID: "aac5cd64-6203-43ee-aa05-d5b8c1e0240a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:52.124617 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.124588 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-dshm" pod "aac5cd64-6203-43ee-aa05-d5b8c1e0240a" (UID: "aac5cd64-6203-43ee-aa05-d5b8c1e0240a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:37:52.124736 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.124711 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-kube-api-access-87zd8" pod "aac5cd64-6203-43ee-aa05-d5b8c1e0240a" (UID: "aac5cd64-6203-43ee-aa05-d5b8c1e0240a"). InnerVolumeSpecName "kube-api-access-87zd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:37:52.124905 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.124883 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-tls-certs" pod "aac5cd64-6203-43ee-aa05-d5b8c1e0240a" (UID: "aac5cd64-6203-43ee-aa05-d5b8c1e0240a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:37:52.206647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.206598 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w"] Oct 30 21:37:52.212943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.212901 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-3b28bc-646fb5564c-rm49w"] Oct 30 21:37:52.223835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.223792 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87zd8\" (UniqueName: \"kubernetes.io/projected/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-kube-api-access-87zd8\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:52.223835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.223836 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:52.224091 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.223850 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:52.224091 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.223862 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:52.224091 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.223872 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aac5cd64-6203-43ee-aa05-d5b8c1e0240a-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:37:52.847848 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.847806 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" event={"ID":"aac5cd64-6203-43ee-aa05-d5b8c1e0240a","Type":"ContainerDied","Data":"b043ed3e47497f0ed58dbbd9c8bcdf2ba692834005232d3961eb439325df18c2"} Oct 30 21:37:52.848086 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.847868 4926 scope.go:122] "RemoveContainer" containerID="5e201a253f5363bfedf22643a9fed3a1fe43e5a4458d9036928bea8ea4b902a8" Oct 30 21:37:52.848086 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.847812 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh" Oct 30 21:37:52.903082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.903024 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh"] Oct 30 21:37:52.913514 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:52.913478 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-abf04b/test-model-abf04b-kserve-c7fcd8789-l7ngh"] Oct 30 21:37:53.016836 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:53.016801 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c" path="/var/lib/kubelet/pods/3f8a51a1-e1d3-4ad1-bdcd-018bbf69528c/volumes" Oct 30 21:37:53.017351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:53.017335 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87506f32-cd27-497b-be39-ef8ae28faef0" path="/var/lib/kubelet/pods/87506f32-cd27-497b-be39-ef8ae28faef0/volumes" Oct 30 21:37:53.017749 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:53.017736 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aac5cd64-6203-43ee-aa05-d5b8c1e0240a" path="/var/lib/kubelet/pods/aac5cd64-6203-43ee-aa05-d5b8c1e0240a/volumes" Oct 30 21:37:53.018289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:53.018269 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0" path="/var/lib/kubelet/pods/c90d209a-8ec6-4ebc-ac5d-d7be15fea2a0/volumes" Oct 30 21:37:53.616712 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:53.616662 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk"] Oct 30 21:37:53.653307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:53.653265 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7"] Oct 30 21:37:56.519487 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.519445 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x"] Oct 30 21:37:56.520062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.520024 4926 cpu_manager.go:414] "containerMap: removing container" podUID="be8482db-1f53-4f48-bf7a-fd17a081c6f9" containerName="maas-api" Oct 30 21:37:56.520062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.520060 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be8482db-1f53-4f48-bf7a-fd17a081c6f9" containerName="maas-api" Oct 30 21:37:56.520275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.520094 4926 cpu_manager.go:414] "containerMap: removing container" podUID="87506f32-cd27-497b-be39-ef8ae28faef0" containerName="maas-api" Oct 30 21:37:56.520275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.520101 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87506f32-cd27-497b-be39-ef8ae28faef0" containerName="maas-api" Oct 30 21:37:56.520275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.520124 4926 cpu_manager.go:414] "containerMap: removing container" podUID="059be9e3-5689-4c00-9554-f7c9f6a2cf59" containerName="revoke-keys" Oct 30 21:37:56.520275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.520132 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="059be9e3-5689-4c00-9554-f7c9f6a2cf59" containerName="revoke-keys" Oct 30 21:37:56.520275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.520150 4926 cpu_manager.go:414] "containerMap: removing container" podUID="aac5cd64-6203-43ee-aa05-d5b8c1e0240a" containerName="main" Oct 30 21:37:56.520275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.520157 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aac5cd64-6203-43ee-aa05-d5b8c1e0240a" containerName="main" Oct 30 21:37:56.520707 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.520304 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="059be9e3-5689-4c00-9554-f7c9f6a2cf59" containerName="revoke-keys" Oct 30 21:37:56.520707 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.520320 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="aac5cd64-6203-43ee-aa05-d5b8c1e0240a" containerName="main" Oct 30 21:37:56.520707 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.520350 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="be8482db-1f53-4f48-bf7a-fd17a081c6f9" containerName="maas-api" Oct 30 21:37:56.520707 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.520366 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="87506f32-cd27-497b-be39-ef8ae28faef0" containerName="maas-api" Oct 30 21:37:56.525645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.525622 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:37:56.543668 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.543636 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x"] Oct 30 21:37:56.548062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.548019 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-908469\"/\"auth-test-model-908469-kserve-self-signed-certs\"" Oct 30 21:37:56.548268 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.548246 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-908469\"/\"default-dockercfg-mvdmg\"" Oct 30 21:37:56.551224 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.551204 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-908469\"/\"kube-root-ca.crt\"" Oct 30 21:37:56.551476 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.551458 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-908469\"/\"openshift-service-ca.crt\"" Oct 30 21:37:56.697673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.697627 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kvpq\" (UniqueName: \"kubernetes.io/projected/32325e5c-9e22-4068-a425-a668cf4b9f04-kube-api-access-5kvpq\") pod \"auth-test-model-908469-kserve-7d97d589d5-kxh2x\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:37:56.697874 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.697761 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-dshm\") pod \"auth-test-model-908469-kserve-7d97d589d5-kxh2x\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:37:56.697874 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.697830 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-home\") pod \"auth-test-model-908469-kserve-7d97d589d5-kxh2x\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:37:56.698027 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.698006 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-tmp-dir\") pod \"auth-test-model-908469-kserve-7d97d589d5-kxh2x\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:37:56.698145 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.698078 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32325e5c-9e22-4068-a425-a668cf4b9f04-tls-certs\") pod \"auth-test-model-908469-kserve-7d97d589d5-kxh2x\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:37:56.698217 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.698188 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-model-cache\") pod \"auth-test-model-908469-kserve-7d97d589d5-kxh2x\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:37:56.799490 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.799379 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kvpq\" (UniqueName: \"kubernetes.io/projected/32325e5c-9e22-4068-a425-a668cf4b9f04-kube-api-access-5kvpq\") pod \"auth-test-model-908469-kserve-7d97d589d5-kxh2x\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:37:56.799697 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.799533 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-dshm\") pod \"auth-test-model-908469-kserve-7d97d589d5-kxh2x\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:37:56.799697 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.799591 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-home\") pod \"auth-test-model-908469-kserve-7d97d589d5-kxh2x\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:37:56.799830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.799732 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-tmp-dir\") pod \"auth-test-model-908469-kserve-7d97d589d5-kxh2x\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:37:56.799830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.799767 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32325e5c-9e22-4068-a425-a668cf4b9f04-tls-certs\") pod \"auth-test-model-908469-kserve-7d97d589d5-kxh2x\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:37:56.799945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.799832 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-model-cache\") pod \"auth-test-model-908469-kserve-7d97d589d5-kxh2x\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:37:56.800197 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.800167 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-home\") pod \"auth-test-model-908469-kserve-7d97d589d5-kxh2x\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:37:56.800310 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.800257 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-model-cache\") pod \"auth-test-model-908469-kserve-7d97d589d5-kxh2x\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:37:56.800611 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.800585 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-tmp-dir\") pod \"auth-test-model-908469-kserve-7d97d589d5-kxh2x\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:37:56.802788 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.802758 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-dshm\") pod \"auth-test-model-908469-kserve-7d97d589d5-kxh2x\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:37:56.803547 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.803507 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32325e5c-9e22-4068-a425-a668cf4b9f04-tls-certs\") pod \"auth-test-model-908469-kserve-7d97d589d5-kxh2x\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:37:56.819074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.819002 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kvpq\" (UniqueName: \"kubernetes.io/projected/32325e5c-9e22-4068-a425-a668cf4b9f04-kube-api-access-5kvpq\") pod \"auth-test-model-908469-kserve-7d97d589d5-kxh2x\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:37:56.842696 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:56.842397 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:37:57.117128 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:57.116637 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss"] Oct 30 21:37:57.117128 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:57.117046 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" podUID="ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" containerName="istio-proxy" containerID="cri-o://e4c60155bf64038f95ee7b41ab80fca992205d1ffc68b88115c5d56f12e0de88" gracePeriod=30 Oct 30 21:37:57.354289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:57.352457 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x"] Oct 30 21:37:57.946741 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:57.946702 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" event={"ID":"32325e5c-9e22-4068-a425-a668cf4b9f04","Type":"ContainerStarted","Data":"7741a61c7a6de7710bf2b59d10eb0f9246f188c4c2fc16413bd52cb2e1e419f7"} Oct 30 21:37:57.946741 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:57.946744 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" event={"ID":"32325e5c-9e22-4068-a425-a668cf4b9f04","Type":"ContainerStarted","Data":"4c3249b7b5d02cdc1f09ce1f140d6e33a54b91d9f5a4b66b2ae1054cb6623163"} Oct 30 21:37:57.985998 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:57.985926 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" podStartSLOduration=1.985912693 podStartE2EDuration="1.985912693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:57.984432863 +0000 UTC m=+2265.105506610" watchObservedRunningTime="2026-10-30 21:37:57.985912693 +0000 UTC m=+2265.106986438" Oct 30 21:37:58.326497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.326418 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j"] Oct 30 21:37:58.330804 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.330778 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.359137 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.359082 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-f5bef0-openshift-default-dockercfg-rgcfl\"" Oct 30 21:37:58.360053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.360006 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv5k4\" (UniqueName: \"kubernetes.io/projected/c58ef83e-f70c-499e-8377-63d4e375601b-kube-api-access-hv5k4\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.360190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.360112 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58ef83e-f70c-499e-8377-63d4e375601b-istiod-ca-cert\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.360271 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.360233 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-workload-certs\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.360327 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.360281 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-istio-data\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.360327 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.360313 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-workload-socket\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.360438 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.360394 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-credential-socket\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.360438 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.360427 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58ef83e-f70c-499e-8377-63d4e375601b-istio-token\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.360555 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.360473 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58ef83e-f70c-499e-8377-63d4e375601b-istio-podinfo\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.360555 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.360512 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-istio-envoy\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.363732 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.363705 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j"] Oct 30 21:37:58.461570 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.461524 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-istio-envoy\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.461817 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.461638 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv5k4\" (UniqueName: \"kubernetes.io/projected/c58ef83e-f70c-499e-8377-63d4e375601b-kube-api-access-hv5k4\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.461817 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.461691 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58ef83e-f70c-499e-8377-63d4e375601b-istiod-ca-cert\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.461817 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.461756 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-workload-certs\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.461817 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.461788 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-istio-data\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.462119 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.461818 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-workload-socket\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.462119 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.461889 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-credential-socket\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.462119 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.461919 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58ef83e-f70c-499e-8377-63d4e375601b-istio-token\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.462119 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.461963 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58ef83e-f70c-499e-8377-63d4e375601b-istio-podinfo\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.462362 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.462320 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-workload-socket\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.462428 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.462391 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-credential-socket\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.462626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.462574 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58ef83e-f70c-499e-8377-63d4e375601b-istiod-ca-cert\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.462626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.462586 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-workload-certs\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.462866 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.462845 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-istio-data\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.464998 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.464977 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58ef83e-f70c-499e-8377-63d4e375601b-istio-podinfo\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.466568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.466541 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-istio-envoy\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.484609 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.484575 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv5k4\" (UniqueName: \"kubernetes.io/projected/c58ef83e-f70c-499e-8377-63d4e375601b-kube-api-access-hv5k4\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.485495 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.485472 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58ef83e-f70c-499e-8377-63d4e375601b-istio-token\") pod \"e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.626639 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.617562 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk" podUID="91ff5943-9467-4ce9-a354-9070c9cef92f" containerName="payload-processing" containerID="cri-o://4d046422c4dc1acaff79f493b824bc152f48e874058e79879c30d9f8357d5116" gracePeriod=25 Oct 30 21:37:58.658070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.656163 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7" podUID="cd180d75-5253-4485-8eed-d47bff0287b2" containerName="payload-pre-processing" containerID="cri-o://ab04111f4cfd5d5f5331394205c7ef8903303aaaf226d326ebbcfa38fe265d60" gracePeriod=25 Oct 30 21:37:58.663289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.662490 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:37:58.843914 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:58.843856 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:37:59.043026 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:59.042993 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j"] Oct 30 21:37:59.043893 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:37:59.043863 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58ef83e_f70c_499e_8377_63d4e375601b.slice/crio-378e929b12b3fc8288c1d0634af0d169ebde36890945ce8796baa8666d6b080b WatchSource:0}: Error finding container 378e929b12b3fc8288c1d0634af0d169ebde36890945ce8796baa8666d6b080b: Status 404 returned error can't find the container with id 378e929b12b3fc8288c1d0634af0d169ebde36890945ce8796baa8666d6b080b Oct 30 21:37:59.047327 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:59.047293 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:37:59.047424 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:59.047364 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:37:59.047424 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:59.047408 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:37:59.966221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:59.966175 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" event={"ID":"c58ef83e-f70c-499e-8377-63d4e375601b","Type":"ContainerStarted","Data":"3c7125df7f2c6e991d35ab60f6ea98d175725029b42cae56451c9824a3de73e4"} Oct 30 21:37:59.966439 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:37:59.966247 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" event={"ID":"c58ef83e-f70c-499e-8377-63d4e375601b","Type":"ContainerStarted","Data":"378e929b12b3fc8288c1d0634af0d169ebde36890945ce8796baa8666d6b080b"} Oct 30 21:38:00.057805 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:00.057733 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" podStartSLOduration=2.057709645 podStartE2EDuration="2.057709645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:00.038364422 +0000 UTC m=+2267.159438170" watchObservedRunningTime="2026-10-30 21:38:00.057709645 +0000 UTC m=+2267.178783392" Oct 30 21:38:00.329822 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:00.329731 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v"] Oct 30 21:38:00.329987 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:00.329964 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" podUID="f0c795d0-9420-4f20-862a-7c09c66bd016" containerName="istio-proxy" containerID="cri-o://40ec18a10f41377c268c1b1d7590119ad4063ceea6d74fdd9ec82f5a5db5a2f9" gracePeriod=30 Oct 30 21:38:00.663209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:00.663107 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:38:00.670426 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:00.670381 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:38:00.978947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:00.978902 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:38:00.980117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:00.980077 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:38:02.816266 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.816238 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:38:02.967859 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.967823 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istiod-ca-cert\") pod \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " Oct 30 21:38:02.968078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.967878 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-workload-socket\") pod \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " Oct 30 21:38:02.968078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.967928 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-kube-api-access-tk67j\" (UniqueName: \"kubernetes.io/projected/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-kube-api-access-tk67j\") pod \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " Oct 30 21:38:02.968078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.967973 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-workload-certs\") pod \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " Oct 30 21:38:02.968255 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.968171 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-credential-socket\") pod \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " Oct 30 21:38:02.968323 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.968299 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-workload-socket" pod "ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" (UID: "ef8d8639-34b4-40d2-ac12-2bdcdcc3544d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:02.968445 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.968424 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-credential-socket" pod "ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" (UID: "ef8d8639-34b4-40d2-ac12-2bdcdcc3544d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:02.968536 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.968451 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-workload-certs" pod "ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" (UID: "ef8d8639-34b4-40d2-ac12-2bdcdcc3544d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:02.968536 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.968474 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-envoy\") pod \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " Oct 30 21:38:02.968632 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.968320 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istiod-ca-cert" pod "ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" (UID: "ef8d8639-34b4-40d2-ac12-2bdcdcc3544d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:02.968632 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.968615 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-token\" (UniqueName: \"kubernetes.io/projected/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-token\") pod \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " Oct 30 21:38:02.968743 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.968650 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-podinfo\") pod \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " Oct 30 21:38:02.968743 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.968714 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-data\") pod \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\" (UID: \"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d\") " Oct 30 21:38:02.969399 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.969349 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-data" pod "ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" (UID: "ef8d8639-34b4-40d2-ac12-2bdcdcc3544d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:02.969399 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.969380 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:02.969608 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.969404 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:02.969608 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.969419 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:02.969608 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.969433 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:02.970763 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.970736 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-kube-api-access-tk67j" pod "ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" (UID: "ef8d8639-34b4-40d2-ac12-2bdcdcc3544d"). InnerVolumeSpecName "kube-api-access-tk67j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:02.971011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.970983 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-token" pod "ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" (UID: "ef8d8639-34b4-40d2-ac12-2bdcdcc3544d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:02.971327 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.971307 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-podinfo" pod "ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" (UID: "ef8d8639-34b4-40d2-ac12-2bdcdcc3544d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:38:02.971558 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:02.971541 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-envoy" pod "ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" (UID: "ef8d8639-34b4-40d2-ac12-2bdcdcc3544d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:03.003276 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:03.003237 4926 generic.go:356] "Generic (PLEG): container finished" podID="ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" containerID="e4c60155bf64038f95ee7b41ab80fca992205d1ffc68b88115c5d56f12e0de88" exitCode=0 Oct 30 21:38:03.003449 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:03.003286 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" Oct 30 21:38:03.003449 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:03.003281 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" event={"ID":"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d","Type":"ContainerDied","Data":"e4c60155bf64038f95ee7b41ab80fca992205d1ffc68b88115c5d56f12e0de88"} Oct 30 21:38:03.003554 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:03.003455 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss" event={"ID":"ef8d8639-34b4-40d2-ac12-2bdcdcc3544d","Type":"ContainerDied","Data":"8bc7539c8dfd02e99d2a596f7e759fb952540ad6c51a95a26f26a0fe9f7f5e62"} Oct 30 21:38:03.003554 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:03.003489 4926 scope.go:122] "RemoveContainer" containerID="e4c60155bf64038f95ee7b41ab80fca992205d1ffc68b88115c5d56f12e0de88" Oct 30 21:38:03.017473 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:03.017453 4926 scope.go:122] "RemoveContainer" containerID="62d4396fc9cec9e296628ccfc5a668a4237aea8fab4eacf7bc264d64fe6d9a4e" Oct 30 21:38:03.029338 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:03.029313 4926 scope.go:122] "RemoveContainer" containerID="e4c60155bf64038f95ee7b41ab80fca992205d1ffc68b88115c5d56f12e0de88" Oct 30 21:38:03.029654 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:38:03.029633 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4c60155bf64038f95ee7b41ab80fca992205d1ffc68b88115c5d56f12e0de88\": container with ID starting with e4c60155bf64038f95ee7b41ab80fca992205d1ffc68b88115c5d56f12e0de88 not found: ID does not exist" containerID="e4c60155bf64038f95ee7b41ab80fca992205d1ffc68b88115c5d56f12e0de88" Oct 30 21:38:03.029734 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:03.029664 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c60155bf64038f95ee7b41ab80fca992205d1ffc68b88115c5d56f12e0de88"} err="failed to get container status \"e4c60155bf64038f95ee7b41ab80fca992205d1ffc68b88115c5d56f12e0de88\": rpc error: code = NotFound desc = could not find container \"e4c60155bf64038f95ee7b41ab80fca992205d1ffc68b88115c5d56f12e0de88\": container with ID starting with e4c60155bf64038f95ee7b41ab80fca992205d1ffc68b88115c5d56f12e0de88 not found: ID does not exist" Oct 30 21:38:03.029734 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:03.029682 4926 scope.go:122] "RemoveContainer" containerID="62d4396fc9cec9e296628ccfc5a668a4237aea8fab4eacf7bc264d64fe6d9a4e" Oct 30 21:38:03.029948 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:38:03.029930 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62d4396fc9cec9e296628ccfc5a668a4237aea8fab4eacf7bc264d64fe6d9a4e\": container with ID starting with 62d4396fc9cec9e296628ccfc5a668a4237aea8fab4eacf7bc264d64fe6d9a4e not found: ID does not exist" containerID="62d4396fc9cec9e296628ccfc5a668a4237aea8fab4eacf7bc264d64fe6d9a4e" Oct 30 21:38:03.029997 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:03.029952 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62d4396fc9cec9e296628ccfc5a668a4237aea8fab4eacf7bc264d64fe6d9a4e"} err="failed to get container status \"62d4396fc9cec9e296628ccfc5a668a4237aea8fab4eacf7bc264d64fe6d9a4e\": rpc error: code = NotFound desc = could not find container \"62d4396fc9cec9e296628ccfc5a668a4237aea8fab4eacf7bc264d64fe6d9a4e\": container with ID starting with 62d4396fc9cec9e296628ccfc5a668a4237aea8fab4eacf7bc264d64fe6d9a4e not found: ID does not exist" Oct 30 21:38:03.058995 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:03.058961 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss"] Oct 30 21:38:03.066282 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:03.066247 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-200d11-openshift-default-5cb455cdfc-6rqss"] Oct 30 21:38:03.071137 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:03.071111 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:03.071137 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:03.071134 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:03.071317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:03.071146 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:03.071317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:03.071158 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:03.071317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:03.071169 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk67j\" (UniqueName: \"kubernetes.io/projected/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d-kube-api-access-tk67j\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:04.182592 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.182545 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv"] Oct 30 21:38:04.183463 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.183431 4926 cpu_manager.go:414] "containerMap: removing container" podUID="ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" containerName="istio-proxy" Oct 30 21:38:04.183463 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.183461 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" containerName="istio-proxy" Oct 30 21:38:04.183671 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.183659 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" containerName="istio-proxy" Oct 30 21:38:04.183734 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.183675 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" containerName="istio-proxy" Oct 30 21:38:04.189836 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.189805 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv" Oct 30 21:38:04.204189 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.204143 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-f5bef0\"" Oct 30 21:38:04.206717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.206680 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv"] Oct 30 21:38:04.210874 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.210844 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-f5bef0-dockercfg-jp7vj\"" Oct 30 21:38:04.308133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.308075 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkqhb\" (UniqueName: \"kubernetes.io/projected/1210d99e-e106-4c6e-8947-cb54d3ec4c2e-kube-api-access-zkqhb\") pod \"payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv\" (UID: \"1210d99e-e106-4c6e-8947-cb54d3ec4c2e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv" Oct 30 21:38:04.308345 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.308224 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1210d99e-e106-4c6e-8947-cb54d3ec4c2e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv\" (UID: \"1210d99e-e106-4c6e-8947-cb54d3ec4c2e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv" Oct 30 21:38:04.412419 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.412372 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkqhb\" (UniqueName: \"kubernetes.io/projected/1210d99e-e106-4c6e-8947-cb54d3ec4c2e-kube-api-access-zkqhb\") pod \"payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv\" (UID: \"1210d99e-e106-4c6e-8947-cb54d3ec4c2e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv" Oct 30 21:38:04.412632 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.412522 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1210d99e-e106-4c6e-8947-cb54d3ec4c2e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv\" (UID: \"1210d99e-e106-4c6e-8947-cb54d3ec4c2e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv" Oct 30 21:38:04.413640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.413611 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1210d99e-e106-4c6e-8947-cb54d3ec4c2e-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv\" (UID: \"1210d99e-e106-4c6e-8947-cb54d3ec4c2e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv" Oct 30 21:38:04.433373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.433278 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkqhb\" (UniqueName: \"kubernetes.io/projected/1210d99e-e106-4c6e-8947-cb54d3ec4c2e-kube-api-access-zkqhb\") pod \"payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv\" (UID: \"1210d99e-e106-4c6e-8947-cb54d3ec4c2e\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv" Oct 30 21:38:04.508871 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.508829 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv" Oct 30 21:38:04.761566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.761475 4926 patch_prober.go:29] interesting pod/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:38:04.761566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.761541 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" podUID="f0c795d0-9420-4f20-862a-7c09c66bd016" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:38:04.828709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.828660 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv"] Oct 30 21:38:04.836293 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:38:04.836252 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1210d99e_e106_4c6e_8947_cb54d3ec4c2e.slice/crio-6b95eb33dd68ff828bd60c79d1c7c4fee4686f936f1f1d6607ba93b6e75d85ff WatchSource:0}: Error finding container 6b95eb33dd68ff828bd60c79d1c7c4fee4686f936f1f1d6607ba93b6e75d85ff: Status 404 returned error can't find the container with id 6b95eb33dd68ff828bd60c79d1c7c4fee4686f936f1f1d6607ba93b6e75d85ff Oct 30 21:38:04.880472 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.880435 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5"] Oct 30 21:38:04.881292 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.881269 4926 cpu_manager.go:414] "containerMap: removing container" podUID="ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" containerName="istio-proxy" Oct 30 21:38:04.881428 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.881293 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" containerName="istio-proxy" Oct 30 21:38:04.886498 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.886460 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:04.903992 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.901711 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-75c17d-openshift-default-dockercfg-48vwh\"" Oct 30 21:38:04.908872 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:04.908838 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5"] Oct 30 21:38:05.018874 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.018791 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef8d8639-34b4-40d2-ac12-2bdcdcc3544d" path="/var/lib/kubelet/pods/ef8d8639-34b4-40d2-ac12-2bdcdcc3544d/volumes" Oct 30 21:38:05.029285 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.029242 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv" event={"ID":"1210d99e-e106-4c6e-8947-cb54d3ec4c2e","Type":"ContainerStarted","Data":"22714bbf960eae8afff4887f37d9d26b2306f61e8f15f438c652665d2c7f8b97"} Oct 30 21:38:05.029434 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.029309 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv" event={"ID":"1210d99e-e106-4c6e-8947-cb54d3ec4c2e","Type":"ContainerStarted","Data":"6b95eb33dd68ff828bd60c79d1c7c4fee4686f936f1f1d6607ba93b6e75d85ff"} Oct 30 21:38:05.029474 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.029448 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv" Oct 30 21:38:05.032008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.031986 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-workload-certs\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.032114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.032023 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsmmz\" (UniqueName: \"kubernetes.io/projected/bb16f268-1822-483a-bdca-87bafd95625e-kube-api-access-bsmmz\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.032174 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.032160 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb16f268-1822-483a-bdca-87bafd95625e-istio-token\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.032219 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.032202 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb16f268-1822-483a-bdca-87bafd95625e-istio-podinfo\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.032254 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.032224 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-istio-envoy\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.032307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.032296 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-credential-socket\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.032357 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.032345 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-istio-data\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.032408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.032398 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb16f268-1822-483a-bdca-87bafd95625e-istiod-ca-cert\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.032485 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.032470 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-workload-socket\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.078343 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.078276 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv" podStartSLOduration=1.07825984 podStartE2EDuration="1.07825984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:05.073764188 +0000 UTC m=+2272.194837933" watchObservedRunningTime="2026-10-30 21:38:05.07825984 +0000 UTC m=+2272.199333585" Oct 30 21:38:05.133358 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.133322 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb16f268-1822-483a-bdca-87bafd95625e-istio-podinfo\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.133562 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.133365 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-istio-envoy\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.133562 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.133408 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-credential-socket\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.133562 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.133437 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-istio-data\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.133562 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.133523 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb16f268-1822-483a-bdca-87bafd95625e-istiod-ca-cert\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.133562 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.133562 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-workload-socket\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.133868 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.133584 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-workload-certs\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.133868 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.133609 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsmmz\" (UniqueName: \"kubernetes.io/projected/bb16f268-1822-483a-bdca-87bafd95625e-kube-api-access-bsmmz\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.133868 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.133702 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb16f268-1822-483a-bdca-87bafd95625e-istio-token\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.134023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.133878 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-credential-socket\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.134023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.133909 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-istio-data\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.134245 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.134222 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-workload-socket\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.134372 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.134349 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-workload-certs\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.135023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.135000 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb16f268-1822-483a-bdca-87bafd95625e-istiod-ca-cert\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.135816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.135791 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-istio-envoy\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.136077 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.136057 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb16f268-1822-483a-bdca-87bafd95625e-istio-podinfo\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.153240 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.153212 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb16f268-1822-483a-bdca-87bafd95625e-istio-token\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.153353 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.153339 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsmmz\" (UniqueName: \"kubernetes.io/projected/bb16f268-1822-483a-bdca-87bafd95625e-kube-api-access-bsmmz\") pod \"e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.225132 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.225087 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:05.571082 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:38:05.570742 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb16f268_1822_483a_bdca_87bafd95625e.slice/crio-e78a1b74667bb13a91d16771fc84ede3429bda5f146796cb2e795b532cf1875d WatchSource:0}: Error finding container e78a1b74667bb13a91d16771fc84ede3429bda5f146796cb2e795b532cf1875d: Status 404 returned error can't find the container with id e78a1b74667bb13a91d16771fc84ede3429bda5f146796cb2e795b532cf1875d Oct 30 21:38:05.577005 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.574509 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:38:05.577005 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.574605 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:38:05.577005 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.574650 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:38:05.577005 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.575081 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5"] Oct 30 21:38:05.679082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.676737 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:38:05.742138 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.742102 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-token\" (UniqueName: \"kubernetes.io/projected/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-token\") pod \"f0c795d0-9420-4f20-862a-7c09c66bd016\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " Oct 30 21:38:05.743547 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.743510 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-credential-socket\") pod \"f0c795d0-9420-4f20-862a-7c09c66bd016\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " Oct 30 21:38:05.743726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.743565 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0c795d0-9420-4f20-862a-7c09c66bd016-kube-api-access-cfdvz\" (UniqueName: \"kubernetes.io/projected/f0c795d0-9420-4f20-862a-7c09c66bd016-kube-api-access-cfdvz\") pod \"f0c795d0-9420-4f20-862a-7c09c66bd016\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " Oct 30 21:38:05.743726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.743671 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0c795d0-9420-4f20-862a-7c09c66bd016-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c795d0-9420-4f20-862a-7c09c66bd016-istiod-ca-cert\") pod \"f0c795d0-9420-4f20-862a-7c09c66bd016\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " Oct 30 21:38:05.743888 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.743762 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-workload-certs\") pod \"f0c795d0-9420-4f20-862a-7c09c66bd016\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " Oct 30 21:38:05.745286 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.744507 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-data\") pod \"f0c795d0-9420-4f20-862a-7c09c66bd016\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " Oct 30 21:38:05.745286 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.744437 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-workload-certs" pod "f0c795d0-9420-4f20-862a-7c09c66bd016" (UID: "f0c795d0-9420-4f20-862a-7c09c66bd016"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:05.745286 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.744562 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-envoy\") pod \"f0c795d0-9420-4f20-862a-7c09c66bd016\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " Oct 30 21:38:05.745286 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.744624 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-podinfo\") pod \"f0c795d0-9420-4f20-862a-7c09c66bd016\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " Oct 30 21:38:05.745286 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.744514 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0c795d0-9420-4f20-862a-7c09c66bd016-istiod-ca-cert" pod "f0c795d0-9420-4f20-862a-7c09c66bd016" (UID: "f0c795d0-9420-4f20-862a-7c09c66bd016"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:05.745286 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.744361 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-credential-socket" pod "f0c795d0-9420-4f20-862a-7c09c66bd016" (UID: "f0c795d0-9420-4f20-862a-7c09c66bd016"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:05.745286 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.744859 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-data" pod "f0c795d0-9420-4f20-862a-7c09c66bd016" (UID: "f0c795d0-9420-4f20-862a-7c09c66bd016"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:05.745643 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.745589 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:05.745643 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.745614 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c795d0-9420-4f20-862a-7c09c66bd016-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:05.745643 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.745628 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:05.745643 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.745640 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:05.749148 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.748387 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-envoy" pod "f0c795d0-9420-4f20-862a-7c09c66bd016" (UID: "f0c795d0-9420-4f20-862a-7c09c66bd016"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:05.749148 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.749085 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-token" pod "f0c795d0-9420-4f20-862a-7c09c66bd016" (UID: "f0c795d0-9420-4f20-862a-7c09c66bd016"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:05.749326 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.749205 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c795d0-9420-4f20-862a-7c09c66bd016-kube-api-access-cfdvz" pod "f0c795d0-9420-4f20-862a-7c09c66bd016" (UID: "f0c795d0-9420-4f20-862a-7c09c66bd016"). InnerVolumeSpecName "kube-api-access-cfdvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:05.752070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.750575 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-podinfo" pod "f0c795d0-9420-4f20-862a-7c09c66bd016" (UID: "f0c795d0-9420-4f20-862a-7c09c66bd016"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:38:05.846619 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.846521 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-workload-socket\") pod \"f0c795d0-9420-4f20-862a-7c09c66bd016\" (UID: \"f0c795d0-9420-4f20-862a-7c09c66bd016\") " Oct 30 21:38:05.846973 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.846950 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-workload-socket" pod "f0c795d0-9420-4f20-862a-7c09c66bd016" (UID: "f0c795d0-9420-4f20-862a-7c09c66bd016"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:05.847192 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.847171 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:05.847232 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.847201 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfdvz\" (UniqueName: \"kubernetes.io/projected/f0c795d0-9420-4f20-862a-7c09c66bd016-kube-api-access-cfdvz\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:05.847232 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.847217 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:05.847315 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.847232 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c795d0-9420-4f20-862a-7c09c66bd016-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:05.847315 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:05.847244 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c795d0-9420-4f20-862a-7c09c66bd016-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:06.040276 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:06.040241 4926 generic.go:356] "Generic (PLEG): container finished" podID="f0c795d0-9420-4f20-862a-7c09c66bd016" containerID="40ec18a10f41377c268c1b1d7590119ad4063ceea6d74fdd9ec82f5a5db5a2f9" exitCode=0 Oct 30 21:38:06.040452 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:06.040332 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" Oct 30 21:38:06.040452 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:06.040327 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" event={"ID":"f0c795d0-9420-4f20-862a-7c09c66bd016","Type":"ContainerDied","Data":"40ec18a10f41377c268c1b1d7590119ad4063ceea6d74fdd9ec82f5a5db5a2f9"} Oct 30 21:38:06.040587 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:06.040494 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v" event={"ID":"f0c795d0-9420-4f20-862a-7c09c66bd016","Type":"ContainerDied","Data":"077b027abbec215331e7b32924f90328744bcc4e133abb130021bc330fa4c96b"} Oct 30 21:38:06.040587 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:06.040524 4926 scope.go:122] "RemoveContainer" containerID="40ec18a10f41377c268c1b1d7590119ad4063ceea6d74fdd9ec82f5a5db5a2f9" Oct 30 21:38:06.042386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:06.042360 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" event={"ID":"bb16f268-1822-483a-bdca-87bafd95625e","Type":"ContainerStarted","Data":"1ac366253e8b8b9f2a35e4ac19df45333893f878710b6218673569bfbec59de0"} Oct 30 21:38:06.042511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:06.042405 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" event={"ID":"bb16f268-1822-483a-bdca-87bafd95625e","Type":"ContainerStarted","Data":"e78a1b74667bb13a91d16771fc84ede3429bda5f146796cb2e795b532cf1875d"} Oct 30 21:38:06.053201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:06.053175 4926 scope.go:122] "RemoveContainer" containerID="40ec18a10f41377c268c1b1d7590119ad4063ceea6d74fdd9ec82f5a5db5a2f9" Oct 30 21:38:06.053536 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:38:06.053516 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ec18a10f41377c268c1b1d7590119ad4063ceea6d74fdd9ec82f5a5db5a2f9\": container with ID starting with 40ec18a10f41377c268c1b1d7590119ad4063ceea6d74fdd9ec82f5a5db5a2f9 not found: ID does not exist" containerID="40ec18a10f41377c268c1b1d7590119ad4063ceea6d74fdd9ec82f5a5db5a2f9" Oct 30 21:38:06.053596 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:06.053545 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ec18a10f41377c268c1b1d7590119ad4063ceea6d74fdd9ec82f5a5db5a2f9"} err="failed to get container status \"40ec18a10f41377c268c1b1d7590119ad4063ceea6d74fdd9ec82f5a5db5a2f9\": rpc error: code = NotFound desc = could not find container \"40ec18a10f41377c268c1b1d7590119ad4063ceea6d74fdd9ec82f5a5db5a2f9\": container with ID starting with 40ec18a10f41377c268c1b1d7590119ad4063ceea6d74fdd9ec82f5a5db5a2f9 not found: ID does not exist" Oct 30 21:38:06.092536 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:06.092494 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v"] Oct 30 21:38:06.102814 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:06.102724 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3b28bc-openshift-default-7b668b8685-4lf7v"] Oct 30 21:38:06.145307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:06.145252 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" podStartSLOduration=2.145232071 podStartE2EDuration="2.145232071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:06.139979199 +0000 UTC m=+2273.261052946" watchObservedRunningTime="2026-10-30 21:38:06.145232071 +0000 UTC m=+2273.266305808" Oct 30 21:38:06.226079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:06.226010 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:06.227507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:06.227479 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 30 21:38:06.227615 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:06.227537 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" podUID="bb16f268-1822-483a-bdca-87bafd95625e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 30 21:38:06.888057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:06.878748 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:38:07.045209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:07.045170 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0c795d0-9420-4f20-862a-7c09c66bd016" path="/var/lib/kubelet/pods/f0c795d0-9420-4f20-862a-7c09c66bd016/volumes" Oct 30 21:38:07.046007 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:07.045985 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:38:07.129060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:07.128608 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:38:07.142716 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:07.142632 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h"] Oct 30 21:38:07.149356 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:07.148901 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b2628a-5rq9h"] Oct 30 21:38:07.227504 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:07.227466 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 30 21:38:07.228139 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:07.228104 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" podUID="bb16f268-1822-483a-bdca-87bafd95625e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 30 21:38:07.516093 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:38:07.516052 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:38:08.225610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:08.225567 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 30 21:38:08.225843 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:08.225630 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" podUID="bb16f268-1822-483a-bdca-87bafd95625e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 30 21:38:08.448146 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:08.448095 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk" podUID="91ff5943-9467-4ce9-a354-9070c9cef92f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:38:08.448146 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:08.448126 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7" podUID="cd180d75-5253-4485-8eed-d47bff0287b2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:38:09.017358 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:09.017322 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345" path="/var/lib/kubelet/pods/8b6b1d8f-f9cf-41ee-bbe4-8b4fc9a7e345/volumes" Oct 30 21:38:09.226191 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:09.226155 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 30 21:38:09.226381 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:09.226212 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" podUID="bb16f268-1822-483a-bdca-87bafd95625e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 30 21:38:10.226540 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:10.226499 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 30 21:38:10.226982 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:10.226580 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" podUID="bb16f268-1822-483a-bdca-87bafd95625e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 30 21:38:11.226200 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:11.226155 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 30 21:38:11.226398 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:11.226224 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" podUID="bb16f268-1822-483a-bdca-87bafd95625e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 30 21:38:12.226342 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:12.226307 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body= Oct 30 21:38:12.226889 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:12.226379 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" podUID="bb16f268-1822-483a-bdca-87bafd95625e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" Oct 30 21:38:13.274876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:13.274840 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:13.276206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:13.276176 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:13.276771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:13.276747 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:38:14.911401 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:14.911361 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd"] Oct 30 21:38:14.912307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:14.912281 4926 cpu_manager.go:414] "containerMap: removing container" podUID="f0c795d0-9420-4f20-862a-7c09c66bd016" containerName="istio-proxy" Oct 30 21:38:14.912373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:14.912311 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0c795d0-9420-4f20-862a-7c09c66bd016" containerName="istio-proxy" Oct 30 21:38:14.912514 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:14.912503 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0c795d0-9420-4f20-862a-7c09c66bd016" containerName="istio-proxy" Oct 30 21:38:14.917814 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:14.917786 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd" Oct 30 21:38:14.962088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:14.954466 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:38:14.962088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:14.954542 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:38:14.991086 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:14.979551 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd"] Oct 30 21:38:15.057600 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.057561 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2a3d20-487a-4b36-91ce-2e54ccb87d24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd\" (UID: \"0f2a3d20-487a-4b36-91ce-2e54ccb87d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd" Oct 30 21:38:15.057930 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.057884 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8fc6\" (UniqueName: \"kubernetes.io/projected/0f2a3d20-487a-4b36-91ce-2e54ccb87d24-kube-api-access-q8fc6\") pod \"maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd\" (UID: \"0f2a3d20-487a-4b36-91ce-2e54ccb87d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd" Oct 30 21:38:15.058596 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.058573 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2a3d20-487a-4b36-91ce-2e54ccb87d24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd\" (UID: \"0f2a3d20-487a-4b36-91ce-2e54ccb87d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd" Oct 30 21:38:15.160099 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.160055 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2a3d20-487a-4b36-91ce-2e54ccb87d24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd\" (UID: \"0f2a3d20-487a-4b36-91ce-2e54ccb87d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd" Oct 30 21:38:15.161345 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.160388 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8fc6\" (UniqueName: \"kubernetes.io/projected/0f2a3d20-487a-4b36-91ce-2e54ccb87d24-kube-api-access-q8fc6\") pod \"maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd\" (UID: \"0f2a3d20-487a-4b36-91ce-2e54ccb87d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd" Oct 30 21:38:15.161345 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.160456 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2a3d20-487a-4b36-91ce-2e54ccb87d24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd\" (UID: \"0f2a3d20-487a-4b36-91ce-2e54ccb87d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd" Oct 30 21:38:15.161345 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.161295 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2a3d20-487a-4b36-91ce-2e54ccb87d24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd\" (UID: \"0f2a3d20-487a-4b36-91ce-2e54ccb87d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd" Oct 30 21:38:15.162908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.162841 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2a3d20-487a-4b36-91ce-2e54ccb87d24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd\" (UID: \"0f2a3d20-487a-4b36-91ce-2e54ccb87d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd" Oct 30 21:38:15.181629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.181594 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8fc6\" (UniqueName: \"kubernetes.io/projected/0f2a3d20-487a-4b36-91ce-2e54ccb87d24-kube-api-access-q8fc6\") pod \"maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd\" (UID: \"0f2a3d20-487a-4b36-91ce-2e54ccb87d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd" Oct 30 21:38:15.280349 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.280304 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd" Oct 30 21:38:15.638466 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.638433 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd"] Oct 30 21:38:15.639845 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:38:15.639810 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2a3d20_487a_4b36_91ce_2e54ccb87d24.slice/crio-50d01f405ac76631999449069205ced5271687596c5c13b6fad985f51710bfa9 WatchSource:0}: Error finding container 50d01f405ac76631999449069205ced5271687596c5c13b6fad985f51710bfa9: Status 404 returned error can't find the container with id 50d01f405ac76631999449069205ced5271687596c5c13b6fad985f51710bfa9 Oct 30 21:38:15.702414 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.702379 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n"] Oct 30 21:38:15.709164 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.709135 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n" Oct 30 21:38:15.736119 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.730131 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n"] Oct 30 21:38:15.871003 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.870960 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f30abe34-8aee-4dce-ac89-155e1a0248bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07367a00-ctv2n\" (UID: \"f30abe34-8aee-4dce-ac89-155e1a0248bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n" Oct 30 21:38:15.871220 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.871201 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f30abe34-8aee-4dce-ac89-155e1a0248bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07367a00-ctv2n\" (UID: \"f30abe34-8aee-4dce-ac89-155e1a0248bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n" Oct 30 21:38:15.871273 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.871240 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zwps\" (UniqueName: \"kubernetes.io/projected/f30abe34-8aee-4dce-ac89-155e1a0248bc-kube-api-access-9zwps\") pod \"maas-api-revoke-sub-07367a00-ctv2n\" (UID: \"f30abe34-8aee-4dce-ac89-155e1a0248bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n" Oct 30 21:38:15.972340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.972305 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f30abe34-8aee-4dce-ac89-155e1a0248bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07367a00-ctv2n\" (UID: \"f30abe34-8aee-4dce-ac89-155e1a0248bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n" Oct 30 21:38:15.972845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.972443 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f30abe34-8aee-4dce-ac89-155e1a0248bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07367a00-ctv2n\" (UID: \"f30abe34-8aee-4dce-ac89-155e1a0248bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n" Oct 30 21:38:15.972845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.972486 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zwps\" (UniqueName: \"kubernetes.io/projected/f30abe34-8aee-4dce-ac89-155e1a0248bc-kube-api-access-9zwps\") pod \"maas-api-revoke-sub-07367a00-ctv2n\" (UID: \"f30abe34-8aee-4dce-ac89-155e1a0248bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n" Oct 30 21:38:15.973204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.973181 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f30abe34-8aee-4dce-ac89-155e1a0248bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07367a00-ctv2n\" (UID: \"f30abe34-8aee-4dce-ac89-155e1a0248bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n" Oct 30 21:38:15.974810 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.974790 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f30abe34-8aee-4dce-ac89-155e1a0248bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07367a00-ctv2n\" (UID: \"f30abe34-8aee-4dce-ac89-155e1a0248bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n" Oct 30 21:38:15.999217 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:15.999186 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zwps\" (UniqueName: \"kubernetes.io/projected/f30abe34-8aee-4dce-ac89-155e1a0248bc-kube-api-access-9zwps\") pod \"maas-api-revoke-sub-07367a00-ctv2n\" (UID: \"f30abe34-8aee-4dce-ac89-155e1a0248bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n" Oct 30 21:38:16.023684 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:16.023645 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n" Oct 30 21:38:16.046096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:16.046064 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv" Oct 30 21:38:16.208045 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:16.207994 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd" event={"ID":"0f2a3d20-487a-4b36-91ce-2e54ccb87d24","Type":"ContainerStarted","Data":"ac77fe218d5e9f4e9031b63926cdd34e6ca8ad945c480799882346de419f6990"} Oct 30 21:38:16.208236 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:16.208104 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd" event={"ID":"0f2a3d20-487a-4b36-91ce-2e54ccb87d24","Type":"ContainerStarted","Data":"50d01f405ac76631999449069205ced5271687596c5c13b6fad985f51710bfa9"} Oct 30 21:38:16.249250 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:16.249133 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd" podStartSLOduration=2.24911673 podStartE2EDuration="2.24911673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:16.247791535 +0000 UTC m=+2283.368865282" watchObservedRunningTime="2026-10-30 21:38:16.24911673 +0000 UTC m=+2283.370190477" Oct 30 21:38:16.351424 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:16.351392 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n"] Oct 30 21:38:16.352970 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:38:16.352935 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf30abe34_8aee_4dce_ac89_155e1a0248bc.slice/crio-77bc0c44addfae308e4cb5acaef4de75b0125d5c4a25faabe021e903eb988b0c WatchSource:0}: Error finding container 77bc0c44addfae308e4cb5acaef4de75b0125d5c4a25faabe021e903eb988b0c: Status 404 returned error can't find the container with id 77bc0c44addfae308e4cb5acaef4de75b0125d5c4a25faabe021e903eb988b0c Oct 30 21:38:17.085931 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:17.085888 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv"] Oct 30 21:38:17.108293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:17.100402 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9477c-6f7sv"] Oct 30 21:38:17.219690 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:17.219637 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n" event={"ID":"f30abe34-8aee-4dce-ac89-155e1a0248bc","Type":"ContainerStarted","Data":"a6a95324e5b584b5e60cc03c3623e91f2f612d744d11d32778ac91e33dda563c"} Oct 30 21:38:17.219918 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:17.219704 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n" event={"ID":"f30abe34-8aee-4dce-ac89-155e1a0248bc","Type":"ContainerStarted","Data":"77bc0c44addfae308e4cb5acaef4de75b0125d5c4a25faabe021e903eb988b0c"} Oct 30 21:38:17.231450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:17.231405 4926 generic.go:356] "Generic (PLEG): container finished" podID="0f2a3d20-487a-4b36-91ce-2e54ccb87d24" containerID="ac77fe218d5e9f4e9031b63926cdd34e6ca8ad945c480799882346de419f6990" exitCode=0 Oct 30 21:38:17.231614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:17.231455 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd" event={"ID":"0f2a3d20-487a-4b36-91ce-2e54ccb87d24","Type":"ContainerDied","Data":"ac77fe218d5e9f4e9031b63926cdd34e6ca8ad945c480799882346de419f6990"} Oct 30 21:38:17.263912 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:17.263850 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n" podStartSLOduration=2.26383052 podStartE2EDuration="2.26383052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:17.260554459 +0000 UTC m=+2284.381628207" watchObservedRunningTime="2026-10-30 21:38:17.26383052 +0000 UTC m=+2284.384904266" Oct 30 21:38:18.254981 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:18.254936 4926 generic.go:356] "Generic (PLEG): container finished" podID="f30abe34-8aee-4dce-ac89-155e1a0248bc" containerID="a6a95324e5b584b5e60cc03c3623e91f2f612d744d11d32778ac91e33dda563c" exitCode=0 Oct 30 21:38:18.255531 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:18.255043 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n" event={"ID":"f30abe34-8aee-4dce-ac89-155e1a0248bc","Type":"ContainerDied","Data":"a6a95324e5b584b5e60cc03c3623e91f2f612d744d11d32778ac91e33dda563c"} Oct 30 21:38:18.447831 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:18.447791 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7" podUID="cd180d75-5253-4485-8eed-d47bff0287b2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:38:18.447831 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:18.447811 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk" podUID="91ff5943-9467-4ce9-a354-9070c9cef92f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:38:18.453484 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:18.453458 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd" Oct 30 21:38:18.525561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:18.525468 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f2a3d20-487a-4b36-91ce-2e54ccb87d24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2a3d20-487a-4b36-91ce-2e54ccb87d24-maas-api-xks-ca\") pod \"0f2a3d20-487a-4b36-91ce-2e54ccb87d24\" (UID: \"0f2a3d20-487a-4b36-91ce-2e54ccb87d24\") " Oct 30 21:38:18.525561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:18.525522 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f2a3d20-487a-4b36-91ce-2e54ccb87d24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2a3d20-487a-4b36-91ce-2e54ccb87d24-maas-api-openshift-ca\") pod \"0f2a3d20-487a-4b36-91ce-2e54ccb87d24\" (UID: \"0f2a3d20-487a-4b36-91ce-2e54ccb87d24\") " Oct 30 21:38:18.525804 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:18.525627 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2a3d20-487a-4b36-91ce-2e54ccb87d24-kube-api-access-q8fc6\" (UniqueName: \"kubernetes.io/projected/0f2a3d20-487a-4b36-91ce-2e54ccb87d24-kube-api-access-q8fc6\") pod \"0f2a3d20-487a-4b36-91ce-2e54ccb87d24\" (UID: \"0f2a3d20-487a-4b36-91ce-2e54ccb87d24\") " Oct 30 21:38:18.526512 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:18.526477 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2a3d20-487a-4b36-91ce-2e54ccb87d24-maas-api-openshift-ca" pod "0f2a3d20-487a-4b36-91ce-2e54ccb87d24" (UID: "0f2a3d20-487a-4b36-91ce-2e54ccb87d24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:18.528786 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:18.528740 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2a3d20-487a-4b36-91ce-2e54ccb87d24-maas-api-xks-ca" pod "0f2a3d20-487a-4b36-91ce-2e54ccb87d24" (UID: "0f2a3d20-487a-4b36-91ce-2e54ccb87d24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:18.528786 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:18.528777 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2a3d20-487a-4b36-91ce-2e54ccb87d24-kube-api-access-q8fc6" pod "0f2a3d20-487a-4b36-91ce-2e54ccb87d24" (UID: "0f2a3d20-487a-4b36-91ce-2e54ccb87d24"). InnerVolumeSpecName "kube-api-access-q8fc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:18.627548 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:18.627504 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f2a3d20-487a-4b36-91ce-2e54ccb87d24-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:18.627548 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:18.627548 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f2a3d20-487a-4b36-91ce-2e54ccb87d24-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:18.627775 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:18.627564 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8fc6\" (UniqueName: \"kubernetes.io/projected/0f2a3d20-487a-4b36-91ce-2e54ccb87d24-kube-api-access-q8fc6\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:18.980353 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:18.980322 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk" Oct 30 21:38:18.993858 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:18.993826 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7" Oct 30 21:38:19.017691 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.017649 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="059be9e3-5689-4c00-9554-f7c9f6a2cf59" path="/var/lib/kubelet/pods/059be9e3-5689-4c00-9554-f7c9f6a2cf59/volumes" Oct 30 21:38:19.042908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.032423 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd180d75-5253-4485-8eed-d47bff0287b2-kube-api-access-d96c2\" (UniqueName: \"kubernetes.io/projected/cd180d75-5253-4485-8eed-d47bff0287b2-kube-api-access-d96c2\") pod \"cd180d75-5253-4485-8eed-d47bff0287b2\" (UID: \"cd180d75-5253-4485-8eed-d47bff0287b2\") " Oct 30 21:38:19.042908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.032626 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd180d75-5253-4485-8eed-d47bff0287b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd180d75-5253-4485-8eed-d47bff0287b2-plugins-config-volume\") pod \"cd180d75-5253-4485-8eed-d47bff0287b2\" (UID: \"cd180d75-5253-4485-8eed-d47bff0287b2\") " Oct 30 21:38:19.042908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.032733 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91ff5943-9467-4ce9-a354-9070c9cef92f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ff5943-9467-4ce9-a354-9070c9cef92f-plugins-config-volume\") pod \"91ff5943-9467-4ce9-a354-9070c9cef92f\" (UID: \"91ff5943-9467-4ce9-a354-9070c9cef92f\") " Oct 30 21:38:19.042908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.032789 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ff5943-9467-4ce9-a354-9070c9cef92f-kube-api-access-5dr27\" (UniqueName: \"kubernetes.io/projected/91ff5943-9467-4ce9-a354-9070c9cef92f-kube-api-access-5dr27\") pod \"91ff5943-9467-4ce9-a354-9070c9cef92f\" (UID: \"91ff5943-9467-4ce9-a354-9070c9cef92f\") " Oct 30 21:38:19.042908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.033155 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd180d75-5253-4485-8eed-d47bff0287b2-plugins-config-volume" pod "cd180d75-5253-4485-8eed-d47bff0287b2" (UID: "cd180d75-5253-4485-8eed-d47bff0287b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:19.042908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.033215 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ff5943-9467-4ce9-a354-9070c9cef92f-plugins-config-volume" pod "91ff5943-9467-4ce9-a354-9070c9cef92f" (UID: "91ff5943-9467-4ce9-a354-9070c9cef92f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:19.042908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.033372 4926 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91ff5943-9467-4ce9-a354-9070c9cef92f-plugins-config-volume\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:19.042908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.033395 4926 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd180d75-5253-4485-8eed-d47bff0287b2-plugins-config-volume\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:19.042908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.037923 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ff5943-9467-4ce9-a354-9070c9cef92f-kube-api-access-5dr27" pod "91ff5943-9467-4ce9-a354-9070c9cef92f" (UID: "91ff5943-9467-4ce9-a354-9070c9cef92f"). InnerVolumeSpecName "kube-api-access-5dr27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:19.044780 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.044656 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd180d75-5253-4485-8eed-d47bff0287b2-kube-api-access-d96c2" pod "cd180d75-5253-4485-8eed-d47bff0287b2" (UID: "cd180d75-5253-4485-8eed-d47bff0287b2"). InnerVolumeSpecName "kube-api-access-d96c2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:19.134945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.134906 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dr27\" (UniqueName: \"kubernetes.io/projected/91ff5943-9467-4ce9-a354-9070c9cef92f-kube-api-access-5dr27\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:19.134945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.134941 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d96c2\" (UniqueName: \"kubernetes.io/projected/cd180d75-5253-4485-8eed-d47bff0287b2-kube-api-access-d96c2\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:19.265220 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.265184 4926 generic.go:356] "Generic (PLEG): container finished" podID="91ff5943-9467-4ce9-a354-9070c9cef92f" containerID="4d046422c4dc1acaff79f493b824bc152f48e874058e79879c30d9f8357d5116" exitCode=0 Oct 30 21:38:19.265673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.265226 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk" Oct 30 21:38:19.265673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.265265 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk" event={"ID":"91ff5943-9467-4ce9-a354-9070c9cef92f","Type":"ContainerDied","Data":"4d046422c4dc1acaff79f493b824bc152f48e874058e79879c30d9f8357d5116"} Oct 30 21:38:19.265673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.265325 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk" event={"ID":"91ff5943-9467-4ce9-a354-9070c9cef92f","Type":"ContainerDied","Data":"cd1b6bb6309a2bf26f9e7ee3e72ac75701df7f5d78bafc8cead2f0868f3c0de7"} Oct 30 21:38:19.265673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.265355 4926 scope.go:122] "RemoveContainer" containerID="4d046422c4dc1acaff79f493b824bc152f48e874058e79879c30d9f8357d5116" Oct 30 21:38:19.267083 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.267053 4926 generic.go:356] "Generic (PLEG): container finished" podID="cd180d75-5253-4485-8eed-d47bff0287b2" containerID="ab04111f4cfd5d5f5331394205c7ef8903303aaaf226d326ebbcfa38fe265d60" exitCode=0 Oct 30 21:38:19.267224 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.267165 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7" Oct 30 21:38:19.267300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.267246 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7" event={"ID":"cd180d75-5253-4485-8eed-d47bff0287b2","Type":"ContainerDied","Data":"ab04111f4cfd5d5f5331394205c7ef8903303aaaf226d326ebbcfa38fe265d60"} Oct 30 21:38:19.267300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.267291 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7" event={"ID":"cd180d75-5253-4485-8eed-d47bff0287b2","Type":"ContainerDied","Data":"0ce110f58ab86a758d88602bee710cd06e6f943808c9a18b04fcf8a359ffe229"} Oct 30 21:38:19.268948 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.268924 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd" event={"ID":"0f2a3d20-487a-4b36-91ce-2e54ccb87d24","Type":"ContainerDied","Data":"50d01f405ac76631999449069205ced5271687596c5c13b6fad985f51710bfa9"} Oct 30 21:38:19.269110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.268967 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50d01f405ac76631999449069205ced5271687596c5c13b6fad985f51710bfa9" Oct 30 21:38:19.269110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.269058 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd" Oct 30 21:38:19.280596 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.280573 4926 scope.go:122] "RemoveContainer" containerID="4d046422c4dc1acaff79f493b824bc152f48e874058e79879c30d9f8357d5116" Oct 30 21:38:19.280927 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:38:19.280904 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d046422c4dc1acaff79f493b824bc152f48e874058e79879c30d9f8357d5116\": container with ID starting with 4d046422c4dc1acaff79f493b824bc152f48e874058e79879c30d9f8357d5116 not found: ID does not exist" containerID="4d046422c4dc1acaff79f493b824bc152f48e874058e79879c30d9f8357d5116" Oct 30 21:38:19.281052 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.280946 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d046422c4dc1acaff79f493b824bc152f48e874058e79879c30d9f8357d5116"} err="failed to get container status \"4d046422c4dc1acaff79f493b824bc152f48e874058e79879c30d9f8357d5116\": rpc error: code = NotFound desc = could not find container \"4d046422c4dc1acaff79f493b824bc152f48e874058e79879c30d9f8357d5116\": container with ID starting with 4d046422c4dc1acaff79f493b824bc152f48e874058e79879c30d9f8357d5116 not found: ID does not exist" Oct 30 21:38:19.281052 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.280966 4926 scope.go:122] "RemoveContainer" containerID="ab04111f4cfd5d5f5331394205c7ef8903303aaaf226d326ebbcfa38fe265d60" Oct 30 21:38:19.296072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.296042 4926 scope.go:122] "RemoveContainer" containerID="ab04111f4cfd5d5f5331394205c7ef8903303aaaf226d326ebbcfa38fe265d60" Oct 30 21:38:19.296412 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:38:19.296392 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab04111f4cfd5d5f5331394205c7ef8903303aaaf226d326ebbcfa38fe265d60\": container with ID starting with ab04111f4cfd5d5f5331394205c7ef8903303aaaf226d326ebbcfa38fe265d60 not found: ID does not exist" containerID="ab04111f4cfd5d5f5331394205c7ef8903303aaaf226d326ebbcfa38fe265d60" Oct 30 21:38:19.296495 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.296424 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab04111f4cfd5d5f5331394205c7ef8903303aaaf226d326ebbcfa38fe265d60"} err="failed to get container status \"ab04111f4cfd5d5f5331394205c7ef8903303aaaf226d326ebbcfa38fe265d60\": rpc error: code = NotFound desc = could not find container \"ab04111f4cfd5d5f5331394205c7ef8903303aaaf226d326ebbcfa38fe265d60\": container with ID starting with ab04111f4cfd5d5f5331394205c7ef8903303aaaf226d326ebbcfa38fe265d60 not found: ID does not exist" Oct 30 21:38:19.319446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.319411 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7"] Oct 30 21:38:19.324521 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.324491 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c9477c-6f4b5bc9-shcq7"] Oct 30 21:38:19.383055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.382817 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk"] Oct 30 21:38:19.390398 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.390363 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c9477c-bff6d4884-l56hk"] Oct 30 21:38:19.420131 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.420103 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n" Oct 30 21:38:19.440981 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.440950 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f30abe34-8aee-4dce-ac89-155e1a0248bc-kube-api-access-9zwps\" (UniqueName: \"kubernetes.io/projected/f30abe34-8aee-4dce-ac89-155e1a0248bc-kube-api-access-9zwps\") pod \"f30abe34-8aee-4dce-ac89-155e1a0248bc\" (UID: \"f30abe34-8aee-4dce-ac89-155e1a0248bc\") " Oct 30 21:38:19.441228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.441104 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f30abe34-8aee-4dce-ac89-155e1a0248bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f30abe34-8aee-4dce-ac89-155e1a0248bc-maas-api-openshift-ca\") pod \"f30abe34-8aee-4dce-ac89-155e1a0248bc\" (UID: \"f30abe34-8aee-4dce-ac89-155e1a0248bc\") " Oct 30 21:38:19.441228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.441141 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f30abe34-8aee-4dce-ac89-155e1a0248bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f30abe34-8aee-4dce-ac89-155e1a0248bc-maas-api-xks-ca\") pod \"f30abe34-8aee-4dce-ac89-155e1a0248bc\" (UID: \"f30abe34-8aee-4dce-ac89-155e1a0248bc\") " Oct 30 21:38:19.441546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.441514 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f30abe34-8aee-4dce-ac89-155e1a0248bc-maas-api-openshift-ca" pod "f30abe34-8aee-4dce-ac89-155e1a0248bc" (UID: "f30abe34-8aee-4dce-ac89-155e1a0248bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:19.441648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.441621 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f30abe34-8aee-4dce-ac89-155e1a0248bc-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:19.443348 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.443317 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f30abe34-8aee-4dce-ac89-155e1a0248bc-maas-api-xks-ca" pod "f30abe34-8aee-4dce-ac89-155e1a0248bc" (UID: "f30abe34-8aee-4dce-ac89-155e1a0248bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:19.443450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.443352 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f30abe34-8aee-4dce-ac89-155e1a0248bc-kube-api-access-9zwps" pod "f30abe34-8aee-4dce-ac89-155e1a0248bc" (UID: "f30abe34-8aee-4dce-ac89-155e1a0248bc"). InnerVolumeSpecName "kube-api-access-9zwps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:19.543009 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.542968 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zwps\" (UniqueName: \"kubernetes.io/projected/f30abe34-8aee-4dce-ac89-155e1a0248bc-kube-api-access-9zwps\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:19.543009 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:19.543006 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f30abe34-8aee-4dce-ac89-155e1a0248bc-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:20.281316 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:20.281285 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n" Oct 30 21:38:20.281316 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:20.281308 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n" event={"ID":"f30abe34-8aee-4dce-ac89-155e1a0248bc","Type":"ContainerDied","Data":"77bc0c44addfae308e4cb5acaef4de75b0125d5c4a25faabe021e903eb988b0c"} Oct 30 21:38:20.281909 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:20.281374 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77bc0c44addfae308e4cb5acaef4de75b0125d5c4a25faabe021e903eb988b0c" Oct 30 21:38:21.017817 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:21.017783 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ff5943-9467-4ce9-a354-9070c9cef92f" path="/var/lib/kubelet/pods/91ff5943-9467-4ce9-a354-9070c9cef92f/volumes" Oct 30 21:38:21.018283 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:21.018269 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd180d75-5253-4485-8eed-d47bff0287b2" path="/var/lib/kubelet/pods/cd180d75-5253-4485-8eed-d47bff0287b2/volumes" Oct 30 21:38:24.230751 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:24.230710 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn"] Oct 30 21:38:24.231348 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:24.230947 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" podUID="c3b985e6-7718-4c16-896b-960d0f1d744e" containerName="istio-proxy" containerID="cri-o://a54b81410563246f57638c188c3b09e679129371df96ee608e58b77f451b885c" gracePeriod=30 Oct 30 21:38:24.247861 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:24.247816 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x"] Oct 30 21:38:24.248167 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:24.248112 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" podUID="61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" containerName="istio-proxy" containerID="cri-o://4a8984badf27ef286f33ed41fe398f5a939b4a8c838e0da9fcf4da9d9e2e4ed2" gracePeriod=30 Oct 30 21:38:25.450462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.450042 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm"] Oct 30 21:38:25.450952 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.450917 4926 cpu_manager.go:414] "containerMap: removing container" podUID="f30abe34-8aee-4dce-ac89-155e1a0248bc" containerName="revoke-keys" Oct 30 21:38:25.450952 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.450940 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f30abe34-8aee-4dce-ac89-155e1a0248bc" containerName="revoke-keys" Oct 30 21:38:25.451111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.450956 4926 cpu_manager.go:414] "containerMap: removing container" podUID="cd180d75-5253-4485-8eed-d47bff0287b2" containerName="payload-pre-processing" Oct 30 21:38:25.451111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.450965 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd180d75-5253-4485-8eed-d47bff0287b2" containerName="payload-pre-processing" Oct 30 21:38:25.451111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.451003 4926 cpu_manager.go:414] "containerMap: removing container" podUID="0f2a3d20-487a-4b36-91ce-2e54ccb87d24" containerName="revoke-keys" Oct 30 21:38:25.451111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.451012 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2a3d20-487a-4b36-91ce-2e54ccb87d24" containerName="revoke-keys" Oct 30 21:38:25.451317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.451126 4926 cpu_manager.go:414] "containerMap: removing container" podUID="91ff5943-9467-4ce9-a354-9070c9cef92f" containerName="payload-processing" Oct 30 21:38:25.451317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.451138 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ff5943-9467-4ce9-a354-9070c9cef92f" containerName="payload-processing" Oct 30 21:38:25.451317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.451298 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2a3d20-487a-4b36-91ce-2e54ccb87d24" containerName="revoke-keys" Oct 30 21:38:25.451469 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.451320 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ff5943-9467-4ce9-a354-9070c9cef92f" containerName="payload-processing" Oct 30 21:38:25.451469 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.451334 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="f30abe34-8aee-4dce-ac89-155e1a0248bc" containerName="revoke-keys" Oct 30 21:38:25.451469 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.451347 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd180d75-5253-4485-8eed-d47bff0287b2" containerName="payload-pre-processing" Oct 30 21:38:25.456735 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.456706 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.488061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.486839 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm"] Oct 30 21:38:25.492184 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.488267 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-66cc54-openshift-default-dockercfg-2mkb6\"" Oct 30 21:38:25.532363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.532322 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-credential-socket\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.532543 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.532372 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-data\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.532543 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.532450 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-podinfo\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.532543 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.532513 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-workload-socket\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.532747 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.532558 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxcxx\" (UniqueName: \"kubernetes.io/projected/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-kube-api-access-wxcxx\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.532747 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.532587 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-token\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.532747 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.532637 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-workload-certs\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.532747 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.532660 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-envoy\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.532960 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.532821 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istiod-ca-cert\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.652068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.649328 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istiod-ca-cert\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.652068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.649433 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-credential-socket\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.652068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.649460 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-data\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.652068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.649521 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-podinfo\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.652068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.649581 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-workload-socket\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.652068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.649627 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxcxx\" (UniqueName: \"kubernetes.io/projected/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-kube-api-access-wxcxx\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.652068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.649658 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-token\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.652068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.649708 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-workload-certs\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.652068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.649732 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-envoy\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.667585 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.667539 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-credential-socket\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.672060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.668315 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istiod-ca-cert\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.672060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.668608 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-data\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.672060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.669153 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-workload-certs\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.691056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.675830 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-podinfo\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.702492 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.693333 4926 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:38:25.702492 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.693408 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" podUID="c3b985e6-7718-4c16-896b-960d0f1d744e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:38:25.703136 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.703093 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-workload-socket\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.716024 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.715012 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxcxx\" (UniqueName: \"kubernetes.io/projected/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-kube-api-access-wxcxx\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.720436 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.720392 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-envoy\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.727759 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.727549 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-token\") pod \"e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:25.802291 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:25.802247 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:26.147112 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:26.147078 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm"] Oct 30 21:38:26.148301 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:38:26.148267 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6202f3_5168_4b2a_a167_c87f1fa98fd1.slice/crio-b76303fdd0b57e63c4b0118d84af091cc36497e3241e91dbfa0cd830f1241aa4 WatchSource:0}: Error finding container b76303fdd0b57e63c4b0118d84af091cc36497e3241e91dbfa0cd830f1241aa4: Status 404 returned error can't find the container with id b76303fdd0b57e63c4b0118d84af091cc36497e3241e91dbfa0cd830f1241aa4 Oct 30 21:38:26.151196 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:26.151157 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:38:26.151296 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:26.151238 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:38:26.151296 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:26.151269 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:38:26.345079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:26.345015 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" event={"ID":"7c6202f3-5168-4b2a-a167-c87f1fa98fd1","Type":"ContainerStarted","Data":"02609b4cdc3a2acc0da054a95129303a39cd410a1141408f1ac4fc04b737f07e"} Oct 30 21:38:26.345267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:26.345100 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" event={"ID":"7c6202f3-5168-4b2a-a167-c87f1fa98fd1","Type":"ContainerStarted","Data":"b76303fdd0b57e63c4b0118d84af091cc36497e3241e91dbfa0cd830f1241aa4"} Oct 30 21:38:26.391056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:26.390702 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" podStartSLOduration=1.3906806999999999 podStartE2EDuration="1.3906807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:26.389491136 +0000 UTC m=+2293.510564882" watchObservedRunningTime="2026-10-30 21:38:26.3906807 +0000 UTC m=+2293.511754447" Oct 30 21:38:26.809053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:26.802897 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:26.896511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:26.896469 4926 patch_prober.go:29] interesting pod/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:38:26.896736 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:26.896537 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" podUID="61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:38:27.810319 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:27.810284 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:28.361653 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:28.361621 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:28.362819 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:28.362800 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:38:29.380630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.380334 4926 generic.go:356] "Generic (PLEG): container finished" podID="c3b985e6-7718-4c16-896b-960d0f1d744e" containerID="a54b81410563246f57638c188c3b09e679129371df96ee608e58b77f451b885c" exitCode=0 Oct 30 21:38:29.380630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.380392 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" event={"ID":"c3b985e6-7718-4c16-896b-960d0f1d744e","Type":"ContainerDied","Data":"a54b81410563246f57638c188c3b09e679129371df96ee608e58b77f451b885c"} Oct 30 21:38:29.383325 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.383294 4926 generic.go:356] "Generic (PLEG): container finished" podID="61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" containerID="4a8984badf27ef286f33ed41fe398f5a939b4a8c838e0da9fcf4da9d9e2e4ed2" exitCode=0 Oct 30 21:38:29.384429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.384173 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" event={"ID":"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7","Type":"ContainerDied","Data":"4a8984badf27ef286f33ed41fe398f5a939b4a8c838e0da9fcf4da9d9e2e4ed2"} Oct 30 21:38:29.384429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.384261 4926 scope.go:122] "RemoveContainer" containerID="fc2716b6916caa2faaea4f9e113d0f21dfd630c4d58ffdf63aac04fd2b32b929" Oct 30 21:38:29.547082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.547052 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:38:29.610133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.608474 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:38:29.639568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.639530 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-envoy\") pod \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " Oct 30 21:38:29.639568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.639573 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b985e6-7718-4c16-896b-960d0f1d744e-kube-api-access-6bw7c\" (UniqueName: \"kubernetes.io/projected/c3b985e6-7718-4c16-896b-960d0f1d744e-kube-api-access-6bw7c\") pod \"c3b985e6-7718-4c16-896b-960d0f1d744e\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " Oct 30 21:38:29.639851 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.639598 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-credential-socket\") pod \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " Oct 30 21:38:29.639851 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.639623 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-token\" (UniqueName: \"kubernetes.io/projected/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-token\") pod \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " Oct 30 21:38:29.639851 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.639662 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-workload-socket\") pod \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " Oct 30 21:38:29.639851 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.639718 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-podinfo\") pod \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " Oct 30 21:38:29.639851 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.639757 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-envoy\") pod \"c3b985e6-7718-4c16-896b-960d0f1d744e\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " Oct 30 21:38:29.639851 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.639806 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-token\" (UniqueName: \"kubernetes.io/projected/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-token\") pod \"c3b985e6-7718-4c16-896b-960d0f1d744e\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " Oct 30 21:38:29.639851 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.639846 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-credential-socket\") pod \"c3b985e6-7718-4c16-896b-960d0f1d744e\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " Oct 30 21:38:29.640289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.639873 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istiod-ca-cert\") pod \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " Oct 30 21:38:29.640289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.639919 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-data\") pod \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " Oct 30 21:38:29.640289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.639963 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-credential-socket" pod "61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" (UID: "61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:29.640289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.640085 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-workload-certs\") pod \"c3b985e6-7718-4c16-896b-960d0f1d744e\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " Oct 30 21:38:29.640289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.640118 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-workload-socket\") pod \"c3b985e6-7718-4c16-896b-960d0f1d744e\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " Oct 30 21:38:29.640289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.640135 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3b985e6-7718-4c16-896b-960d0f1d744e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b985e6-7718-4c16-896b-960d0f1d744e-istiod-ca-cert\") pod \"c3b985e6-7718-4c16-896b-960d0f1d744e\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " Oct 30 21:38:29.640289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.640153 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-workload-certs\") pod \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " Oct 30 21:38:29.640289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.640181 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-data\") pod \"c3b985e6-7718-4c16-896b-960d0f1d744e\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " Oct 30 21:38:29.640289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.640205 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-kube-api-access-g64fr\" (UniqueName: \"kubernetes.io/projected/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-kube-api-access-g64fr\") pod \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\" (UID: \"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7\") " Oct 30 21:38:29.640289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.640237 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-podinfo\") pod \"c3b985e6-7718-4c16-896b-960d0f1d744e\" (UID: \"c3b985e6-7718-4c16-896b-960d0f1d744e\") " Oct 30 21:38:29.640289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.640284 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-workload-socket" pod "61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" (UID: "61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:29.640958 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.640759 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:29.640958 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.640782 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:29.641311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.641189 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-workload-certs" pod "61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" (UID: "61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:29.641508 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.641480 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-workload-socket" pod "c3b985e6-7718-4c16-896b-960d0f1d744e" (UID: "c3b985e6-7718-4c16-896b-960d0f1d744e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:29.641809 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.641785 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istiod-ca-cert" pod "61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" (UID: "61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:29.641910 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.641825 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b985e6-7718-4c16-896b-960d0f1d744e-istiod-ca-cert" pod "c3b985e6-7718-4c16-896b-960d0f1d744e" (UID: "c3b985e6-7718-4c16-896b-960d0f1d744e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:29.641910 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.641883 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-workload-certs" pod "c3b985e6-7718-4c16-896b-960d0f1d744e" (UID: "c3b985e6-7718-4c16-896b-960d0f1d744e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:29.642073 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.641988 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-data" pod "c3b985e6-7718-4c16-896b-960d0f1d744e" (UID: "c3b985e6-7718-4c16-896b-960d0f1d744e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:29.642334 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.642314 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-data" pod "61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" (UID: "61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:29.642459 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.642442 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-credential-socket" pod "c3b985e6-7718-4c16-896b-960d0f1d744e" (UID: "c3b985e6-7718-4c16-896b-960d0f1d744e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:29.644295 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.644269 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-podinfo" pod "c3b985e6-7718-4c16-896b-960d0f1d744e" (UID: "c3b985e6-7718-4c16-896b-960d0f1d744e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:38:29.644486 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.644433 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-token" pod "61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" (UID: "61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:29.644486 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.644457 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b985e6-7718-4c16-896b-960d0f1d744e-kube-api-access-6bw7c" pod "c3b985e6-7718-4c16-896b-960d0f1d744e" (UID: "c3b985e6-7718-4c16-896b-960d0f1d744e"). InnerVolumeSpecName "kube-api-access-6bw7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:29.644486 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.644462 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-envoy" pod "61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" (UID: "61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:29.644486 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.644476 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-token" pod "c3b985e6-7718-4c16-896b-960d0f1d744e" (UID: "c3b985e6-7718-4c16-896b-960d0f1d744e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:29.644675 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.644507 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-envoy" pod "c3b985e6-7718-4c16-896b-960d0f1d744e" (UID: "c3b985e6-7718-4c16-896b-960d0f1d744e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:29.645248 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.645229 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-kube-api-access-g64fr" pod "61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" (UID: "61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7"). InnerVolumeSpecName "kube-api-access-g64fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:29.646146 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.646119 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-podinfo" pod "61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" (UID: "61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:38:29.741688 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.741649 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:29.741688 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.741686 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:29.741947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.741700 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3b985e6-7718-4c16-896b-960d0f1d744e-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:29.741947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.741712 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:29.741947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.741723 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:29.741947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.741734 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g64fr\" (UniqueName: \"kubernetes.io/projected/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-kube-api-access-g64fr\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:29.741947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.741748 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:29.741947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.741760 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:29.741947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.741772 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bw7c\" (UniqueName: \"kubernetes.io/projected/c3b985e6-7718-4c16-896b-960d0f1d744e-kube-api-access-6bw7c\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:29.741947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.741783 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:29.741947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.741794 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:29.741947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.741805 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:29.741947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.741815 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3b985e6-7718-4c16-896b-960d0f1d744e-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:29.741947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.741826 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3b985e6-7718-4c16-896b-960d0f1d744e-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:29.741947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.741838 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:29.741947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:29.741851 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:30.396055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:30.395912 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" Oct 30 21:38:30.396055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:30.395936 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x" event={"ID":"61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7","Type":"ContainerDied","Data":"37a5ec3ba8979b71986d592b915e4639291fff78b7a68950878024b0a15a86ec"} Oct 30 21:38:30.396055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:30.396001 4926 scope.go:122] "RemoveContainer" containerID="4a8984badf27ef286f33ed41fe398f5a939b4a8c838e0da9fcf4da9d9e2e4ed2" Oct 30 21:38:30.398321 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:30.398300 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" Oct 30 21:38:30.398641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:30.398303 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn" event={"ID":"c3b985e6-7718-4c16-896b-960d0f1d744e","Type":"ContainerDied","Data":"0b71674d0e50b9f30474e18cacec9d7c7a47f0d9aae4d6f1542240ed4e822c91"} Oct 30 21:38:30.412730 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:30.412709 4926 scope.go:122] "RemoveContainer" containerID="a54b81410563246f57638c188c3b09e679129371df96ee608e58b77f451b885c" Oct 30 21:38:30.473081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:30.473044 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x"] Oct 30 21:38:30.483784 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:30.483748 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-zkh4x"] Oct 30 21:38:30.527749 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:30.527675 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn"] Oct 30 21:38:30.540690 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:30.540639 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c9477c-openshift-default-548875cd79-bqstn"] Oct 30 21:38:31.019646 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:31.019609 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" path="/var/lib/kubelet/pods/61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7/volumes" Oct 30 21:38:31.020198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:31.020178 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3b985e6-7718-4c16-896b-960d0f1d744e" path="/var/lib/kubelet/pods/c3b985e6-7718-4c16-896b-960d0f1d744e/volumes" Oct 30 21:38:34.828648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:34.828611 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb"] Oct 30 21:38:34.829267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:34.829247 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c3b985e6-7718-4c16-896b-960d0f1d744e" containerName="istio-proxy" Oct 30 21:38:34.829331 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:34.829269 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3b985e6-7718-4c16-896b-960d0f1d744e" containerName="istio-proxy" Oct 30 21:38:34.829331 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:34.829295 4926 cpu_manager.go:414] "containerMap: removing container" podUID="61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" containerName="istio-proxy" Oct 30 21:38:34.829331 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:34.829301 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" containerName="istio-proxy" Oct 30 21:38:34.829426 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:34.829403 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3b985e6-7718-4c16-896b-960d0f1d744e" containerName="istio-proxy" Oct 30 21:38:34.829426 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:34.829414 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" containerName="istio-proxy" Oct 30 21:38:34.834375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:34.834352 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb" Oct 30 21:38:34.852144 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:34.852106 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb"] Oct 30 21:38:34.856969 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:34.856937 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:38:34.857183 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:34.857043 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:38:34.947118 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:34.947077 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc629bb-3e16-4b37-a39e-8ae97e732e84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfb3b2f3-94gjb\" (UID: \"ffc629bb-3e16-4b37-a39e-8ae97e732e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb" Oct 30 21:38:34.947354 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:34.947258 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc629bb-3e16-4b37-a39e-8ae97e732e84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfb3b2f3-94gjb\" (UID: \"ffc629bb-3e16-4b37-a39e-8ae97e732e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb" Oct 30 21:38:34.947354 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:34.947313 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h864b\" (UniqueName: \"kubernetes.io/projected/ffc629bb-3e16-4b37-a39e-8ae97e732e84-kube-api-access-h864b\") pod \"maas-api-revoke-sub-bfb3b2f3-94gjb\" (UID: \"ffc629bb-3e16-4b37-a39e-8ae97e732e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb" Oct 30 21:38:35.049176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:35.049138 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc629bb-3e16-4b37-a39e-8ae97e732e84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfb3b2f3-94gjb\" (UID: \"ffc629bb-3e16-4b37-a39e-8ae97e732e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb" Oct 30 21:38:35.049368 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:35.049238 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc629bb-3e16-4b37-a39e-8ae97e732e84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfb3b2f3-94gjb\" (UID: \"ffc629bb-3e16-4b37-a39e-8ae97e732e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb" Oct 30 21:38:35.049368 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:35.049277 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h864b\" (UniqueName: \"kubernetes.io/projected/ffc629bb-3e16-4b37-a39e-8ae97e732e84-kube-api-access-h864b\") pod \"maas-api-revoke-sub-bfb3b2f3-94gjb\" (UID: \"ffc629bb-3e16-4b37-a39e-8ae97e732e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb" Oct 30 21:38:35.050008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:35.049986 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc629bb-3e16-4b37-a39e-8ae97e732e84-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfb3b2f3-94gjb\" (UID: \"ffc629bb-3e16-4b37-a39e-8ae97e732e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb" Oct 30 21:38:35.051648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:35.051628 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc629bb-3e16-4b37-a39e-8ae97e732e84-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfb3b2f3-94gjb\" (UID: \"ffc629bb-3e16-4b37-a39e-8ae97e732e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb" Oct 30 21:38:35.075011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:35.074971 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h864b\" (UniqueName: \"kubernetes.io/projected/ffc629bb-3e16-4b37-a39e-8ae97e732e84-kube-api-access-h864b\") pod \"maas-api-revoke-sub-bfb3b2f3-94gjb\" (UID: \"ffc629bb-3e16-4b37-a39e-8ae97e732e84\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb" Oct 30 21:38:35.149935 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:35.149832 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb" Oct 30 21:38:35.492931 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:35.492903 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb"] Oct 30 21:38:35.495746 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:38:35.495700 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc629bb_3e16_4b37_a39e_8ae97e732e84.slice/crio-c3386532cb4bd291190ac8d5d74f2d643d2d3a847514ff21002af6e8af5e7087 WatchSource:0}: Error finding container c3386532cb4bd291190ac8d5d74f2d643d2d3a847514ff21002af6e8af5e7087: Status 404 returned error can't find the container with id c3386532cb4bd291190ac8d5d74f2d643d2d3a847514ff21002af6e8af5e7087 Oct 30 21:38:36.484872 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:36.484835 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb" event={"ID":"ffc629bb-3e16-4b37-a39e-8ae97e732e84","Type":"ContainerStarted","Data":"bf4cdd9ed92c64d220a702d0e7201624eb6f07b437a903c6b38c8401fe8b4727"} Oct 30 21:38:36.488334 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:36.484894 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb" event={"ID":"ffc629bb-3e16-4b37-a39e-8ae97e732e84","Type":"ContainerStarted","Data":"c3386532cb4bd291190ac8d5d74f2d643d2d3a847514ff21002af6e8af5e7087"} Oct 30 21:38:36.527968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:36.527907 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb" podStartSLOduration=2.5278867160000003 podStartE2EDuration="2.527886716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:36.522954205 +0000 UTC m=+2303.644027953" watchObservedRunningTime="2026-10-30 21:38:36.527886716 +0000 UTC m=+2303.648960467" Oct 30 21:38:37.489752 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:38:37.489708 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:38:37.499597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:37.499565 4926 generic.go:356] "Generic (PLEG): container finished" podID="ffc629bb-3e16-4b37-a39e-8ae97e732e84" containerID="bf4cdd9ed92c64d220a702d0e7201624eb6f07b437a903c6b38c8401fe8b4727" exitCode=0 Oct 30 21:38:37.499761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:37.499694 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb" event={"ID":"ffc629bb-3e16-4b37-a39e-8ae97e732e84","Type":"ContainerDied","Data":"bf4cdd9ed92c64d220a702d0e7201624eb6f07b437a903c6b38c8401fe8b4727"} Oct 30 21:38:38.704135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:38.704100 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb" Oct 30 21:38:38.815804 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:38.815708 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc629bb-3e16-4b37-a39e-8ae97e732e84-kube-api-access-h864b\" (UniqueName: \"kubernetes.io/projected/ffc629bb-3e16-4b37-a39e-8ae97e732e84-kube-api-access-h864b\") pod \"ffc629bb-3e16-4b37-a39e-8ae97e732e84\" (UID: \"ffc629bb-3e16-4b37-a39e-8ae97e732e84\") " Oct 30 21:38:38.815804 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:38.815792 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc629bb-3e16-4b37-a39e-8ae97e732e84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc629bb-3e16-4b37-a39e-8ae97e732e84-maas-api-xks-ca\") pod \"ffc629bb-3e16-4b37-a39e-8ae97e732e84\" (UID: \"ffc629bb-3e16-4b37-a39e-8ae97e732e84\") " Oct 30 21:38:38.815989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:38.815907 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc629bb-3e16-4b37-a39e-8ae97e732e84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc629bb-3e16-4b37-a39e-8ae97e732e84-maas-api-openshift-ca\") pod \"ffc629bb-3e16-4b37-a39e-8ae97e732e84\" (UID: \"ffc629bb-3e16-4b37-a39e-8ae97e732e84\") " Oct 30 21:38:38.816357 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:38.816321 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc629bb-3e16-4b37-a39e-8ae97e732e84-maas-api-openshift-ca" pod "ffc629bb-3e16-4b37-a39e-8ae97e732e84" (UID: "ffc629bb-3e16-4b37-a39e-8ae97e732e84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:38.816498 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:38.816457 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc629bb-3e16-4b37-a39e-8ae97e732e84-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:38.818177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:38.818147 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc629bb-3e16-4b37-a39e-8ae97e732e84-kube-api-access-h864b" pod "ffc629bb-3e16-4b37-a39e-8ae97e732e84" (UID: "ffc629bb-3e16-4b37-a39e-8ae97e732e84"). InnerVolumeSpecName "kube-api-access-h864b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:38.818422 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:38.818397 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc629bb-3e16-4b37-a39e-8ae97e732e84-maas-api-xks-ca" pod "ffc629bb-3e16-4b37-a39e-8ae97e732e84" (UID: "ffc629bb-3e16-4b37-a39e-8ae97e732e84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:38.917763 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:38.917726 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h864b\" (UniqueName: \"kubernetes.io/projected/ffc629bb-3e16-4b37-a39e-8ae97e732e84-kube-api-access-h864b\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:38.917763 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:38.917757 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc629bb-3e16-4b37-a39e-8ae97e732e84-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:39.521867 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:39.521830 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb" Oct 30 21:38:39.521867 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:39.521849 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb" event={"ID":"ffc629bb-3e16-4b37-a39e-8ae97e732e84","Type":"ContainerDied","Data":"c3386532cb4bd291190ac8d5d74f2d643d2d3a847514ff21002af6e8af5e7087"} Oct 30 21:38:39.522134 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:39.521913 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3386532cb4bd291190ac8d5d74f2d643d2d3a847514ff21002af6e8af5e7087" Oct 30 21:38:40.611377 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:40.611332 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:38:43.228989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:43.228946 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd"] Oct 30 21:38:43.239624 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:43.239590 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-abf04b-rqcnd"] Oct 30 21:38:45.016991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:45.016956 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f2a3d20-487a-4b36-91ce-2e54ccb87d24" path="/var/lib/kubelet/pods/0f2a3d20-487a-4b36-91ce-2e54ccb87d24/volumes" Oct 30 21:38:46.258439 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:46.258393 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt"] Oct 30 21:38:46.259240 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:46.259218 4926 cpu_manager.go:414] "containerMap: removing container" podUID="ffc629bb-3e16-4b37-a39e-8ae97e732e84" containerName="revoke-keys" Oct 30 21:38:46.259240 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:46.259238 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc629bb-3e16-4b37-a39e-8ae97e732e84" containerName="revoke-keys" Oct 30 21:38:46.259440 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:46.259262 4926 cpu_manager.go:414] "containerMap: removing container" podUID="61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" containerName="istio-proxy" Oct 30 21:38:46.259440 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:46.259269 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" containerName="istio-proxy" Oct 30 21:38:46.259440 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:46.259431 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc629bb-3e16-4b37-a39e-8ae97e732e84" containerName="revoke-keys" Oct 30 21:38:46.259630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:46.259445 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="61a7b62e-0eb4-43a8-aae4-24a2d8ab5fc7" containerName="istio-proxy" Oct 30 21:38:46.264645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:46.264621 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt" Oct 30 21:38:46.274182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:46.274152 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt"] Oct 30 21:38:46.279253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:46.279220 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:38:46.289543 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:46.289503 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:38:46.344526 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:46.344484 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1027909d-33fd-48d8-999a-3e361dde86b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt\" (UID: \"1027909d-33fd-48d8-999a-3e361dde86b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt" Oct 30 21:38:46.344718 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:46.344696 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1027909d-33fd-48d8-999a-3e361dde86b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt\" (UID: \"1027909d-33fd-48d8-999a-3e361dde86b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt" Oct 30 21:38:46.344793 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:46.344748 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99tkq\" (UniqueName: \"kubernetes.io/projected/1027909d-33fd-48d8-999a-3e361dde86b2-kube-api-access-99tkq\") pod \"maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt\" (UID: \"1027909d-33fd-48d8-999a-3e361dde86b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt" Oct 30 21:38:46.447348 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:46.447292 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1027909d-33fd-48d8-999a-3e361dde86b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt\" (UID: \"1027909d-33fd-48d8-999a-3e361dde86b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt" Oct 30 21:38:46.447672 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:46.447647 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99tkq\" (UniqueName: \"kubernetes.io/projected/1027909d-33fd-48d8-999a-3e361dde86b2-kube-api-access-99tkq\") pod \"maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt\" (UID: \"1027909d-33fd-48d8-999a-3e361dde86b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt" Oct 30 21:38:46.448002 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:46.447983 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1027909d-33fd-48d8-999a-3e361dde86b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt\" (UID: \"1027909d-33fd-48d8-999a-3e361dde86b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt" Oct 30 21:38:46.448306 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:46.448277 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1027909d-33fd-48d8-999a-3e361dde86b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt\" (UID: \"1027909d-33fd-48d8-999a-3e361dde86b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt" Oct 30 21:38:46.451093 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:46.451067 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1027909d-33fd-48d8-999a-3e361dde86b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt\" (UID: \"1027909d-33fd-48d8-999a-3e361dde86b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt" Oct 30 21:38:46.470759 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:46.470727 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99tkq\" (UniqueName: \"kubernetes.io/projected/1027909d-33fd-48d8-999a-3e361dde86b2-kube-api-access-99tkq\") pod \"maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt\" (UID: \"1027909d-33fd-48d8-999a-3e361dde86b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt" Oct 30 21:38:46.597560 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:46.597477 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt" Oct 30 21:38:46.914893 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:46.914858 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt"] Oct 30 21:38:46.919611 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:38:46.919173 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1027909d_33fd_48d8_999a_3e361dde86b2.slice/crio-72638b38b8c3e34c7cbf2d658c9a2bf67df85c7990645af9fa5df5248905bc75 WatchSource:0}: Error finding container 72638b38b8c3e34c7cbf2d658c9a2bf67df85c7990645af9fa5df5248905bc75: Status 404 returned error can't find the container with id 72638b38b8c3e34c7cbf2d658c9a2bf67df85c7990645af9fa5df5248905bc75 Oct 30 21:38:47.612252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:47.612147 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt" event={"ID":"1027909d-33fd-48d8-999a-3e361dde86b2","Type":"ContainerStarted","Data":"525c404701b1ecd1b51542cfdbf2ddb2b4ad77df5d1052e1cd6d0f8bed27277f"} Oct 30 21:38:47.612252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:47.612215 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt" event={"ID":"1027909d-33fd-48d8-999a-3e361dde86b2","Type":"ContainerStarted","Data":"72638b38b8c3e34c7cbf2d658c9a2bf67df85c7990645af9fa5df5248905bc75"} Oct 30 21:38:47.652262 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:47.652205 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt" podStartSLOduration=1.652190013 podStartE2EDuration="1.652190013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:47.650890708 +0000 UTC m=+2314.771964454" watchObservedRunningTime="2026-10-30 21:38:47.652190013 +0000 UTC m=+2314.773263758" Oct 30 21:38:48.621638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:48.621602 4926 generic.go:356] "Generic (PLEG): container finished" podID="1027909d-33fd-48d8-999a-3e361dde86b2" containerID="525c404701b1ecd1b51542cfdbf2ddb2b4ad77df5d1052e1cd6d0f8bed27277f" exitCode=0 Oct 30 21:38:48.622187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:48.621648 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt" event={"ID":"1027909d-33fd-48d8-999a-3e361dde86b2","Type":"ContainerDied","Data":"525c404701b1ecd1b51542cfdbf2ddb2b4ad77df5d1052e1cd6d0f8bed27277f"} Oct 30 21:38:49.798311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:49.798282 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt" Oct 30 21:38:49.899157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:49.899021 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1027909d-33fd-48d8-999a-3e361dde86b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1027909d-33fd-48d8-999a-3e361dde86b2-maas-api-xks-ca\") pod \"1027909d-33fd-48d8-999a-3e361dde86b2\" (UID: \"1027909d-33fd-48d8-999a-3e361dde86b2\") " Oct 30 21:38:49.899157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:49.899132 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1027909d-33fd-48d8-999a-3e361dde86b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1027909d-33fd-48d8-999a-3e361dde86b2-maas-api-openshift-ca\") pod \"1027909d-33fd-48d8-999a-3e361dde86b2\" (UID: \"1027909d-33fd-48d8-999a-3e361dde86b2\") " Oct 30 21:38:49.899382 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:49.899182 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1027909d-33fd-48d8-999a-3e361dde86b2-kube-api-access-99tkq\" (UniqueName: \"kubernetes.io/projected/1027909d-33fd-48d8-999a-3e361dde86b2-kube-api-access-99tkq\") pod \"1027909d-33fd-48d8-999a-3e361dde86b2\" (UID: \"1027909d-33fd-48d8-999a-3e361dde86b2\") " Oct 30 21:38:49.899647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:49.899620 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1027909d-33fd-48d8-999a-3e361dde86b2-maas-api-openshift-ca" pod "1027909d-33fd-48d8-999a-3e361dde86b2" (UID: "1027909d-33fd-48d8-999a-3e361dde86b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:49.899920 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:49.899896 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1027909d-33fd-48d8-999a-3e361dde86b2-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:49.901500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:49.901478 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1027909d-33fd-48d8-999a-3e361dde86b2-maas-api-xks-ca" pod "1027909d-33fd-48d8-999a-3e361dde86b2" (UID: "1027909d-33fd-48d8-999a-3e361dde86b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:49.901746 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:49.901713 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1027909d-33fd-48d8-999a-3e361dde86b2-kube-api-access-99tkq" pod "1027909d-33fd-48d8-999a-3e361dde86b2" (UID: "1027909d-33fd-48d8-999a-3e361dde86b2"). InnerVolumeSpecName "kube-api-access-99tkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:50.001761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:50.001717 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1027909d-33fd-48d8-999a-3e361dde86b2-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:50.001761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:50.001754 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99tkq\" (UniqueName: \"kubernetes.io/projected/1027909d-33fd-48d8-999a-3e361dde86b2-kube-api-access-99tkq\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:50.641689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:50.641651 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt" Oct 30 21:38:50.641689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:50.641674 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt" event={"ID":"1027909d-33fd-48d8-999a-3e361dde86b2","Type":"ContainerDied","Data":"72638b38b8c3e34c7cbf2d658c9a2bf67df85c7990645af9fa5df5248905bc75"} Oct 30 21:38:50.641970 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:50.641746 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72638b38b8c3e34c7cbf2d658c9a2bf67df85c7990645af9fa5df5248905bc75" Oct 30 21:38:50.826415 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:50.826369 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9"] Oct 30 21:38:50.827438 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:50.826638 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" podUID="12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" containerName="istio-proxy" containerID="cri-o://c6d339e651788f5bde1866968ea47719d5e5a50fa7d5a2cf955a77a3db54fd5e" gracePeriod=30 Oct 30 21:38:53.229246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:53.229201 4926 patch_prober.go:29] interesting pod/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:38:53.229715 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:53.229275 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" podUID="12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:38:56.115968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.115936 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:38:56.185964 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.185901 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-podinfo\") pod \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " Oct 30 21:38:56.185964 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.185962 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istiod-ca-cert\") pod \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " Oct 30 21:38:56.186252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.186084 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-workload-certs\") pod \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " Oct 30 21:38:56.186328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.186292 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-kube-api-access-h8l2z\" (UniqueName: \"kubernetes.io/projected/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-kube-api-access-h8l2z\") pod \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " Oct 30 21:38:56.186373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.186327 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-token\" (UniqueName: \"kubernetes.io/projected/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-token\") pod \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " Oct 30 21:38:56.186373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.186360 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-envoy\") pod \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " Oct 30 21:38:56.186450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.186389 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-workload-socket\") pod \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " Oct 30 21:38:56.186491 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.186447 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-data\") pod \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " Oct 30 21:38:56.186491 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.186476 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-credential-socket\") pod \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\" (UID: \"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4\") " Oct 30 21:38:56.187547 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.187514 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-workload-socket" pod "12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" (UID: "12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:56.187804 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.187542 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-credential-socket" pod "12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" (UID: "12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:56.187804 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.187765 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-workload-certs" pod "12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" (UID: "12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:56.187804 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.187773 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-data" pod "12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" (UID: "12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:56.188023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.187914 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istiod-ca-cert" pod "12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" (UID: "12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:56.192223 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.192190 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-podinfo" pod "12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" (UID: "12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:38:56.192354 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.192230 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-token" pod "12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" (UID: "12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:56.192354 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.192284 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-kube-api-access-h8l2z" pod "12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" (UID: "12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4"). InnerVolumeSpecName "kube-api-access-h8l2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:56.192354 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.192300 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-envoy" pod "12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" (UID: "12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:56.288840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.288801 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8l2z\" (UniqueName: \"kubernetes.io/projected/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-kube-api-access-h8l2z\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:56.288840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.288841 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:56.289078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.288856 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:56.289078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.288869 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:56.289078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.288882 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:56.289078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.288896 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:56.289078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.288910 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:56.289078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.288925 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:56.289078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.288937 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:38:56.704432 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.704394 4926 generic.go:356] "Generic (PLEG): container finished" podID="12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" containerID="c6d339e651788f5bde1866968ea47719d5e5a50fa7d5a2cf955a77a3db54fd5e" exitCode=0 Oct 30 21:38:56.704643 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.704469 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" Oct 30 21:38:56.704643 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.704479 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" event={"ID":"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4","Type":"ContainerDied","Data":"c6d339e651788f5bde1866968ea47719d5e5a50fa7d5a2cf955a77a3db54fd5e"} Oct 30 21:38:56.704643 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.704548 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9" event={"ID":"12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4","Type":"ContainerDied","Data":"fb5e8e4bd3a26b1852c909ecb395b9fef4c7dc5bb94e5156b05a2e504fa1f4de"} Oct 30 21:38:56.704643 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.704578 4926 scope.go:122] "RemoveContainer" containerID="c6d339e651788f5bde1866968ea47719d5e5a50fa7d5a2cf955a77a3db54fd5e" Oct 30 21:38:56.719139 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.719113 4926 scope.go:122] "RemoveContainer" containerID="0a21174f122abf4b93eef30871d437bc2a4f7bf191945c9aaa103461c74a43da" Oct 30 21:38:56.730868 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.730842 4926 scope.go:122] "RemoveContainer" containerID="c6d339e651788f5bde1866968ea47719d5e5a50fa7d5a2cf955a77a3db54fd5e" Oct 30 21:38:56.731237 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:38:56.731211 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6d339e651788f5bde1866968ea47719d5e5a50fa7d5a2cf955a77a3db54fd5e\": container with ID starting with c6d339e651788f5bde1866968ea47719d5e5a50fa7d5a2cf955a77a3db54fd5e not found: ID does not exist" containerID="c6d339e651788f5bde1866968ea47719d5e5a50fa7d5a2cf955a77a3db54fd5e" Oct 30 21:38:56.731332 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.731248 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6d339e651788f5bde1866968ea47719d5e5a50fa7d5a2cf955a77a3db54fd5e"} err="failed to get container status \"c6d339e651788f5bde1866968ea47719d5e5a50fa7d5a2cf955a77a3db54fd5e\": rpc error: code = NotFound desc = could not find container \"c6d339e651788f5bde1866968ea47719d5e5a50fa7d5a2cf955a77a3db54fd5e\": container with ID starting with c6d339e651788f5bde1866968ea47719d5e5a50fa7d5a2cf955a77a3db54fd5e not found: ID does not exist" Oct 30 21:38:56.731332 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.731268 4926 scope.go:122] "RemoveContainer" containerID="0a21174f122abf4b93eef30871d437bc2a4f7bf191945c9aaa103461c74a43da" Oct 30 21:38:56.731531 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:38:56.731511 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a21174f122abf4b93eef30871d437bc2a4f7bf191945c9aaa103461c74a43da\": container with ID starting with 0a21174f122abf4b93eef30871d437bc2a4f7bf191945c9aaa103461c74a43da not found: ID does not exist" containerID="0a21174f122abf4b93eef30871d437bc2a4f7bf191945c9aaa103461c74a43da" Oct 30 21:38:56.731574 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.731540 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a21174f122abf4b93eef30871d437bc2a4f7bf191945c9aaa103461c74a43da"} err="failed to get container status \"0a21174f122abf4b93eef30871d437bc2a4f7bf191945c9aaa103461c74a43da\": rpc error: code = NotFound desc = could not find container \"0a21174f122abf4b93eef30871d437bc2a4f7bf191945c9aaa103461c74a43da\": container with ID starting with 0a21174f122abf4b93eef30871d437bc2a4f7bf191945c9aaa103461c74a43da not found: ID does not exist" Oct 30 21:38:56.759400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.759359 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9"] Oct 30 21:38:56.769290 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:56.769254 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-abf04b-openshift-default-74ccb66d7f-glxm9"] Oct 30 21:38:57.018548 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:38:57.018456 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" path="/var/lib/kubelet/pods/12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4/volumes" Oct 30 21:39:07.486642 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:39:07.486593 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:39:15.027704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:15.027659 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9"] Oct 30 21:39:15.028319 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:15.028185 4926 cpu_manager.go:414] "containerMap: removing container" podUID="1027909d-33fd-48d8-999a-3e361dde86b2" containerName="revoke-keys" Oct 30 21:39:15.028319 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:15.028199 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1027909d-33fd-48d8-999a-3e361dde86b2" containerName="revoke-keys" Oct 30 21:39:15.028319 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:15.028212 4926 cpu_manager.go:414] "containerMap: removing container" podUID="12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" containerName="istio-proxy" Oct 30 21:39:15.028319 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:15.028218 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" containerName="istio-proxy" Oct 30 21:39:15.028559 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:15.028327 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" containerName="istio-proxy" Oct 30 21:39:15.028559 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:15.028337 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="1027909d-33fd-48d8-999a-3e361dde86b2" containerName="revoke-keys" Oct 30 21:39:15.028559 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:15.028344 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" containerName="istio-proxy" Oct 30 21:39:15.032640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:15.032251 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9" Oct 30 21:39:15.037220 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:15.037079 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9"] Oct 30 21:39:15.064096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:15.063490 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-889195-dockercfg-52cw9\"" Oct 30 21:39:15.064096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:15.063551 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-889195\"" Oct 30 21:39:15.261056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:15.154596 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57902142-df3e-4114-ac6c-7d42a300f2f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9\" (UID: \"57902142-df3e-4114-ac6c-7d42a300f2f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9" Oct 30 21:39:15.267057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:15.261473 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4vjq\" (UniqueName: \"kubernetes.io/projected/57902142-df3e-4114-ac6c-7d42a300f2f9-kube-api-access-s4vjq\") pod \"payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9\" (UID: \"57902142-df3e-4114-ac6c-7d42a300f2f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9" Oct 30 21:39:15.392107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:15.370895 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57902142-df3e-4114-ac6c-7d42a300f2f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9\" (UID: \"57902142-df3e-4114-ac6c-7d42a300f2f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9" Oct 30 21:39:15.393570 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:15.393537 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4vjq\" (UniqueName: \"kubernetes.io/projected/57902142-df3e-4114-ac6c-7d42a300f2f9-kube-api-access-s4vjq\") pod \"payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9\" (UID: \"57902142-df3e-4114-ac6c-7d42a300f2f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9" Oct 30 21:39:15.394310 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:15.391055 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57902142-df3e-4114-ac6c-7d42a300f2f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9\" (UID: \"57902142-df3e-4114-ac6c-7d42a300f2f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9" Oct 30 21:39:15.419175 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:15.419084 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4vjq\" (UniqueName: \"kubernetes.io/projected/57902142-df3e-4114-ac6c-7d42a300f2f9-kube-api-access-s4vjq\") pod \"payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9\" (UID: \"57902142-df3e-4114-ac6c-7d42a300f2f9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9" Oct 30 21:39:15.655182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:15.655101 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9" Oct 30 21:39:16.112964 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:16.112930 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9"] Oct 30 21:39:16.113541 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:39:16.113374 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57902142_df3e_4114_ac6c_7d42a300f2f9.slice/crio-80284b8323e2ebba43f4f5115ae76f6de225502e541039dc3948f17ab3c02061 WatchSource:0}: Error finding container 80284b8323e2ebba43f4f5115ae76f6de225502e541039dc3948f17ab3c02061: Status 404 returned error can't find the container with id 80284b8323e2ebba43f4f5115ae76f6de225502e541039dc3948f17ab3c02061 Oct 30 21:39:16.981122 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:16.981085 4926 scope.go:122] "RemoveContainer" containerID="5b6c7cfe0cf24edd5499a5250f3933498800cd1342032db868d3521f7a603a57" Oct 30 21:39:17.056684 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:17.053854 4926 scope.go:122] "RemoveContainer" containerID="2800712563c8ea00d81a4bdb57683fcd9aae6322a8bc677fac24ee22cafbe612" Oct 30 21:39:17.167680 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:17.167636 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9" event={"ID":"57902142-df3e-4114-ac6c-7d42a300f2f9","Type":"ContainerStarted","Data":"bd88847d4fd0f826b96b52a1e19ed25743f182c7920f92094816cb9f1eceb002"} Oct 30 21:39:17.169052 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:17.168253 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9" event={"ID":"57902142-df3e-4114-ac6c-7d42a300f2f9","Type":"ContainerStarted","Data":"80284b8323e2ebba43f4f5115ae76f6de225502e541039dc3948f17ab3c02061"} Oct 30 21:39:17.169563 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:17.169537 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9" Oct 30 21:39:17.405761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:17.399634 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9" podStartSLOduration=3.399610464 podStartE2EDuration="3.399610464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:17.337148591 +0000 UTC m=+2344.458222337" watchObservedRunningTime="2026-10-30 21:39:17.399610464 +0000 UTC m=+2344.520684210" Oct 30 21:39:17.502765 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:17.502690 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt"] Oct 30 21:39:17.526054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:17.522940 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-67f8e0-lzbqt"] Oct 30 21:39:19.020384 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:19.020344 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1027909d-33fd-48d8-999a-3e361dde86b2" path="/var/lib/kubelet/pods/1027909d-33fd-48d8-999a-3e361dde86b2/volumes" Oct 30 21:39:22.221664 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:22.221625 4926 generic.go:356] "Generic (PLEG): container finished" podID="c58ef83e-f70c-499e-8377-63d4e375601b" containerID="3c7125df7f2c6e991d35ab60f6ea98d175725029b42cae56451c9824a3de73e4" exitCode=0 Oct 30 21:39:22.222159 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:22.221700 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" event={"ID":"c58ef83e-f70c-499e-8377-63d4e375601b","Type":"ContainerDied","Data":"3c7125df7f2c6e991d35ab60f6ea98d175725029b42cae56451c9824a3de73e4"} Oct 30 21:39:22.222159 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:22.222152 4926 scope.go:122] "RemoveContainer" containerID="3c7125df7f2c6e991d35ab60f6ea98d175725029b42cae56451c9824a3de73e4" Oct 30 21:39:22.223814 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:22.223782 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:22.223892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:22.223855 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:22.223936 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:22.223889 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:23.238459 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:23.238411 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" event={"ID":"c58ef83e-f70c-499e-8377-63d4e375601b","Type":"ContainerStarted","Data":"d70e82987d246b50bf5b4d97b4c3dfe5a2180da88e2126101024c4c63250570c"} Oct 30 21:39:23.663257 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:23.663161 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:39:23.663454 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:23.663377 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 21:39:23.663454 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:23.663433 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" podUID="c58ef83e-f70c-499e-8377-63d4e375601b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 21:39:24.664244 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:24.664205 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 21:39:24.665746 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:24.665703 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" podUID="c58ef83e-f70c-499e-8377-63d4e375601b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 21:39:24.836831 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:24.836016 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb"] Oct 30 21:39:24.836831 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:24.836289 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" podUID="eedfd657-1ae2-4911-b92d-a78c44831f18" containerName="istio-proxy" containerID="cri-o://26ed5007a17b3dbe488a558e6701e5178cb0f8c8ed620666efd7895bd146e578" gracePeriod=30 Oct 30 21:39:24.990116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:24.986189 4926 patch_prober.go:29] interesting pod/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:39:24.990116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:24.986273 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" podUID="eedfd657-1ae2-4911-b92d-a78c44831f18" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:39:25.414324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:25.414292 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l"] Oct 30 21:39:25.414877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:25.414863 4926 cpu_manager.go:414] "containerMap: removing container" podUID="12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" containerName="istio-proxy" Oct 30 21:39:25.414927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:25.414878 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12f6bdd5-2b23-4b38-a5d1-2b5fc498c9c4" containerName="istio-proxy" Oct 30 21:39:25.417492 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:25.417471 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l" Oct 30 21:39:25.487063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:25.478611 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:39:25.504061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:25.487522 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:39:25.510501 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:25.510334 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l"] Oct 30 21:39:25.611706 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:25.611666 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhlmk\" (UniqueName: \"kubernetes.io/projected/95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d-kube-api-access-xhlmk\") pod \"maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l\" (UID: \"95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l" Oct 30 21:39:25.611706 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:25.611715 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l\" (UID: \"95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l" Oct 30 21:39:25.612074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:25.612021 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l\" (UID: \"95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l" Oct 30 21:39:25.664100 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:25.664016 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 21:39:25.664285 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:25.664110 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" podUID="c58ef83e-f70c-499e-8377-63d4e375601b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 21:39:25.713713 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:25.712782 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l\" (UID: \"95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l" Oct 30 21:39:25.713713 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:25.712856 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlmk\" (UniqueName: \"kubernetes.io/projected/95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d-kube-api-access-xhlmk\") pod \"maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l\" (UID: \"95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l" Oct 30 21:39:25.713713 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:25.712878 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l\" (UID: \"95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l" Oct 30 21:39:25.713713 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:25.713630 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l\" (UID: \"95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l" Oct 30 21:39:25.716562 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:25.716531 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l\" (UID: \"95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l" Oct 30 21:39:25.733655 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:25.733623 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhlmk\" (UniqueName: \"kubernetes.io/projected/95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d-kube-api-access-xhlmk\") pod \"maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l\" (UID: \"95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l" Oct 30 21:39:25.756525 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:25.756492 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l" Oct 30 21:39:26.108071 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:26.108026 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l"] Oct 30 21:39:26.108761 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:39:26.108729 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b5d5eb_8e5e_4fcd_b6bc_0aa00c65f28d.slice/crio-d61777b31e28b26a2de61e14294a7d7b2ac428e2274e08b18d187d823d6d3249 WatchSource:0}: Error finding container d61777b31e28b26a2de61e14294a7d7b2ac428e2274e08b18d187d823d6d3249: Status 404 returned error can't find the container with id d61777b31e28b26a2de61e14294a7d7b2ac428e2274e08b18d187d823d6d3249 Oct 30 21:39:26.263163 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:26.263123 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l" event={"ID":"95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d","Type":"ContainerStarted","Data":"d61777b31e28b26a2de61e14294a7d7b2ac428e2274e08b18d187d823d6d3249"} Oct 30 21:39:26.664238 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:26.664200 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 21:39:26.664426 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:26.664258 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" podUID="c58ef83e-f70c-499e-8377-63d4e375601b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 21:39:27.301606 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:27.301562 4926 generic.go:356] "Generic (PLEG): container finished" podID="95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d" containerID="5f1470e758efb85fc2619ded37d6f9542eebd76345f2aaef5b98844e028334e8" exitCode=0 Oct 30 21:39:27.301821 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:27.301760 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l" event={"ID":"95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d","Type":"ContainerDied","Data":"5f1470e758efb85fc2619ded37d6f9542eebd76345f2aaef5b98844e028334e8"} Oct 30 21:39:27.664130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:27.664096 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 21:39:27.664292 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:27.664164 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" podUID="c58ef83e-f70c-499e-8377-63d4e375601b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 21:39:28.629380 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:28.625425 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l" Oct 30 21:39:28.647296 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:28.647265 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d-maas-api-xks-ca\") pod \"95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d\" (UID: \"95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d\") " Oct 30 21:39:28.647468 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:28.647320 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d-maas-api-openshift-ca\") pod \"95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d\" (UID: \"95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d\") " Oct 30 21:39:28.647468 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:28.647385 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d-kube-api-access-xhlmk\" (UniqueName: \"kubernetes.io/projected/95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d-kube-api-access-xhlmk\") pod \"95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d\" (UID: \"95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d\") " Oct 30 21:39:28.648350 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:28.648321 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d-maas-api-openshift-ca" pod "95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d" (UID: "95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:28.650370 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:28.649962 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d-kube-api-access-xhlmk" pod "95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d" (UID: "95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d"). InnerVolumeSpecName "kube-api-access-xhlmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:28.650370 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:28.650014 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d-maas-api-xks-ca" pod "95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d" (UID: "95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:28.663821 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:28.663779 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Oct 30 21:39:28.663939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:28.663845 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" podUID="c58ef83e-f70c-499e-8377-63d4e375601b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Oct 30 21:39:28.749353 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:28.749304 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:39:28.749353 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:28.749359 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:39:28.749576 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:28.749370 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhlmk\" (UniqueName: \"kubernetes.io/projected/95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d-kube-api-access-xhlmk\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:39:29.192888 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:29.192853 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9" Oct 30 21:39:29.325064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:29.324965 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l" Oct 30 21:39:29.325064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:29.324994 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l" event={"ID":"95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d","Type":"ContainerDied","Data":"d61777b31e28b26a2de61e14294a7d7b2ac428e2274e08b18d187d823d6d3249"} Oct 30 21:39:29.325064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:29.325062 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d61777b31e28b26a2de61e14294a7d7b2ac428e2274e08b18d187d823d6d3249" Oct 30 21:39:29.668921 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:29.668886 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:39:29.669469 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:29.669363 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:39:29.670177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:29.670152 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:39:30.156829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.156802 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:39:30.267165 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.267064 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-workload-certs\") pod \"eedfd657-1ae2-4911-b92d-a78c44831f18\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " Oct 30 21:39:30.267165 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.267135 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-envoy\") pod \"eedfd657-1ae2-4911-b92d-a78c44831f18\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " Oct 30 21:39:30.267413 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.267208 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-token\" (UniqueName: \"kubernetes.io/projected/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-token\") pod \"eedfd657-1ae2-4911-b92d-a78c44831f18\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " Oct 30 21:39:30.267413 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.267278 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-workload-socket\") pod \"eedfd657-1ae2-4911-b92d-a78c44831f18\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " Oct 30 21:39:30.267413 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.267318 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-data\") pod \"eedfd657-1ae2-4911-b92d-a78c44831f18\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " Oct 30 21:39:30.267413 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.267337 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-workload-certs" pod "eedfd657-1ae2-4911-b92d-a78c44831f18" (UID: "eedfd657-1ae2-4911-b92d-a78c44831f18"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:39:30.267413 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.267348 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-credential-socket\") pod \"eedfd657-1ae2-4911-b92d-a78c44831f18\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " Oct 30 21:39:30.267685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.267425 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eedfd657-1ae2-4911-b92d-a78c44831f18-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eedfd657-1ae2-4911-b92d-a78c44831f18-istiod-ca-cert\") pod \"eedfd657-1ae2-4911-b92d-a78c44831f18\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " Oct 30 21:39:30.267685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.267631 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-data" pod "eedfd657-1ae2-4911-b92d-a78c44831f18" (UID: "eedfd657-1ae2-4911-b92d-a78c44831f18"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:39:30.267685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.267648 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-credential-socket" pod "eedfd657-1ae2-4911-b92d-a78c44831f18" (UID: "eedfd657-1ae2-4911-b92d-a78c44831f18"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:39:30.267927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.267888 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eedfd657-1ae2-4911-b92d-a78c44831f18-istiod-ca-cert" pod "eedfd657-1ae2-4911-b92d-a78c44831f18" (UID: "eedfd657-1ae2-4911-b92d-a78c44831f18"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:30.267927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.267888 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-workload-socket" pod "eedfd657-1ae2-4911-b92d-a78c44831f18" (UID: "eedfd657-1ae2-4911-b92d-a78c44831f18"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:39:30.268023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.267988 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-podinfo\") pod \"eedfd657-1ae2-4911-b92d-a78c44831f18\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " Oct 30 21:39:30.268107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.268074 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eedfd657-1ae2-4911-b92d-a78c44831f18-kube-api-access-jzwts\" (UniqueName: \"kubernetes.io/projected/eedfd657-1ae2-4911-b92d-a78c44831f18-kube-api-access-jzwts\") pod \"eedfd657-1ae2-4911-b92d-a78c44831f18\" (UID: \"eedfd657-1ae2-4911-b92d-a78c44831f18\") " Oct 30 21:39:30.268670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.268643 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:39:30.268670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.268672 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:39:30.268873 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.268687 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:39:30.268873 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.268711 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eedfd657-1ae2-4911-b92d-a78c44831f18-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:39:30.268873 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.268727 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:39:30.269802 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.269771 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-token" pod "eedfd657-1ae2-4911-b92d-a78c44831f18" (UID: "eedfd657-1ae2-4911-b92d-a78c44831f18"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:30.269931 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.269829 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-envoy" pod "eedfd657-1ae2-4911-b92d-a78c44831f18" (UID: "eedfd657-1ae2-4911-b92d-a78c44831f18"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:39:30.270564 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.270534 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-podinfo" pod "eedfd657-1ae2-4911-b92d-a78c44831f18" (UID: "eedfd657-1ae2-4911-b92d-a78c44831f18"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:39:30.270679 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.270608 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eedfd657-1ae2-4911-b92d-a78c44831f18-kube-api-access-jzwts" pod "eedfd657-1ae2-4911-b92d-a78c44831f18" (UID: "eedfd657-1ae2-4911-b92d-a78c44831f18"). InnerVolumeSpecName "kube-api-access-jzwts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:30.335164 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.335127 4926 generic.go:356] "Generic (PLEG): container finished" podID="eedfd657-1ae2-4911-b92d-a78c44831f18" containerID="26ed5007a17b3dbe488a558e6701e5178cb0f8c8ed620666efd7895bd146e578" exitCode=0 Oct 30 21:39:30.335164 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.335175 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" Oct 30 21:39:30.335408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.335211 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" event={"ID":"eedfd657-1ae2-4911-b92d-a78c44831f18","Type":"ContainerDied","Data":"26ed5007a17b3dbe488a558e6701e5178cb0f8c8ed620666efd7895bd146e578"} Oct 30 21:39:30.335408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.335294 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb" event={"ID":"eedfd657-1ae2-4911-b92d-a78c44831f18","Type":"ContainerDied","Data":"66bd96caf09ff99f62b2ac0b5932bdc8e8e303b95d392b0f214d781429483cbd"} Oct 30 21:39:30.335408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.335323 4926 scope.go:122] "RemoveContainer" containerID="26ed5007a17b3dbe488a558e6701e5178cb0f8c8ed620666efd7895bd146e578" Oct 30 21:39:30.347905 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.347882 4926 scope.go:122] "RemoveContainer" containerID="26ed5007a17b3dbe488a558e6701e5178cb0f8c8ed620666efd7895bd146e578" Oct 30 21:39:30.348278 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:39:30.348256 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26ed5007a17b3dbe488a558e6701e5178cb0f8c8ed620666efd7895bd146e578\": container with ID starting with 26ed5007a17b3dbe488a558e6701e5178cb0f8c8ed620666efd7895bd146e578 not found: ID does not exist" containerID="26ed5007a17b3dbe488a558e6701e5178cb0f8c8ed620666efd7895bd146e578" Oct 30 21:39:30.348356 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.348288 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26ed5007a17b3dbe488a558e6701e5178cb0f8c8ed620666efd7895bd146e578"} err="failed to get container status \"26ed5007a17b3dbe488a558e6701e5178cb0f8c8ed620666efd7895bd146e578\": rpc error: code = NotFound desc = could not find container \"26ed5007a17b3dbe488a558e6701e5178cb0f8c8ed620666efd7895bd146e578\": container with ID starting with 26ed5007a17b3dbe488a558e6701e5178cb0f8c8ed620666efd7895bd146e578 not found: ID does not exist" Oct 30 21:39:30.370681 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.370570 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:39:30.370681 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.370600 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:39:30.370681 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.370616 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eedfd657-1ae2-4911-b92d-a78c44831f18-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:39:30.370681 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.370630 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzwts\" (UniqueName: \"kubernetes.io/projected/eedfd657-1ae2-4911-b92d-a78c44831f18-kube-api-access-jzwts\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:39:30.402403 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.402364 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb"] Oct 30 21:39:30.408084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:30.407855 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-67f8e0-openshift-default-954489f5f-kzctb"] Oct 30 21:39:31.020248 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:31.020209 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eedfd657-1ae2-4911-b92d-a78c44831f18" path="/var/lib/kubelet/pods/eedfd657-1ae2-4911-b92d-a78c44831f18/volumes" Oct 30 21:39:35.959986 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:35.959949 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5"] Oct 30 21:39:35.960777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:35.960754 4926 cpu_manager.go:414] "containerMap: removing container" podUID="eedfd657-1ae2-4911-b92d-a78c44831f18" containerName="istio-proxy" Oct 30 21:39:35.960777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:35.960777 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eedfd657-1ae2-4911-b92d-a78c44831f18" containerName="istio-proxy" Oct 30 21:39:35.960964 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:35.960815 4926 cpu_manager.go:414] "containerMap: removing container" podUID="95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d" containerName="revoke-keys" Oct 30 21:39:35.960964 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:35.960823 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d" containerName="revoke-keys" Oct 30 21:39:35.963237 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:35.961001 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="eedfd657-1ae2-4911-b92d-a78c44831f18" containerName="istio-proxy" Oct 30 21:39:35.963237 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:35.961018 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d" containerName="revoke-keys" Oct 30 21:39:35.977465 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:35.965368 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:35.995736 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:35.995697 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5"] Oct 30 21:39:36.000679 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.000646 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-9cd83f-openshift-default-dockercfg-gz5mt\"" Oct 30 21:39:36.054638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.053633 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-data\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.054638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.053743 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44fcef78-1e16-4d54-9306-52f006c0bb9f-istiod-ca-cert\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.054638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.053827 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-token\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.054638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.053869 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-workload-certs\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.054638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.054113 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-podinfo\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.054638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.054205 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-workload-socket\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.054638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.054234 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-envoy\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.054638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.054416 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-credential-socket\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.054638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.054497 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d77cw\" (UniqueName: \"kubernetes.io/projected/44fcef78-1e16-4d54-9306-52f006c0bb9f-kube-api-access-d77cw\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.155829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.155777 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44fcef78-1e16-4d54-9306-52f006c0bb9f-istiod-ca-cert\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.156068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.155882 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-token\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.156068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.155924 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-workload-certs\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.156068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.155982 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-podinfo\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.156068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.156049 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-workload-socket\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.156311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.156075 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-envoy\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.156311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.156135 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-credential-socket\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.156311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.156182 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d77cw\" (UniqueName: \"kubernetes.io/projected/44fcef78-1e16-4d54-9306-52f006c0bb9f-kube-api-access-d77cw\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.156311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.156281 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-data\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.156825 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.156796 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-data\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.157518 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.157489 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44fcef78-1e16-4d54-9306-52f006c0bb9f-istiod-ca-cert\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.158134 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.158110 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-workload-certs\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.163748 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.163721 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-podinfo\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.173110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.170965 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-credential-socket\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.173304 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.171505 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-workload-socket\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.197056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.191692 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-envoy\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.228886 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.228808 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d77cw\" (UniqueName: \"kubernetes.io/projected/44fcef78-1e16-4d54-9306-52f006c0bb9f-kube-api-access-d77cw\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.237995 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.237650 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-token\") pod \"e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.310651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.310345 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:36.641877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.641845 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5"] Oct 30 21:39:36.642797 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:39:36.642771 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44fcef78_1e16_4d54_9306_52f006c0bb9f.slice/crio-f26d782a4a27587dd2d4edf618dd10cb79ebf977e3469e825d1ecc252541338b WatchSource:0}: Error finding container f26d782a4a27587dd2d4edf618dd10cb79ebf977e3469e825d1ecc252541338b: Status 404 returned error can't find the container with id f26d782a4a27587dd2d4edf618dd10cb79ebf977e3469e825d1ecc252541338b Oct 30 21:39:36.645945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.645908 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:36.646092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.645996 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:36.646171 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:36.646088 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:37.407727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:37.407679 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" event={"ID":"44fcef78-1e16-4d54-9306-52f006c0bb9f","Type":"ContainerStarted","Data":"078e4011583f679da1086c5dbc33fd6e4e37f7bd60d39bc2f56dd5e2ed56107c"} Oct 30 21:39:37.408206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:37.407777 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" event={"ID":"44fcef78-1e16-4d54-9306-52f006c0bb9f","Type":"ContainerStarted","Data":"f26d782a4a27587dd2d4edf618dd10cb79ebf977e3469e825d1ecc252541338b"} Oct 30 21:39:37.466499 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:37.466430 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" podStartSLOduration=2.466415531 podStartE2EDuration="2.466415531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:37.462670939 +0000 UTC m=+2364.583744684" watchObservedRunningTime="2026-10-30 21:39:37.466415531 +0000 UTC m=+2364.587489276" Oct 30 21:39:37.488982 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:39:37.488938 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:39:38.311623 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:38.311577 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:38.317965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:38.317935 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:38.417941 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:38.417903 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:38.419149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:38.419123 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:39:47.116100 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.116064 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm"] Oct 30 21:39:47.120637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.120612 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.147811 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.147774 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-95692f-openshift-default-dockercfg-q7k6z\"" Oct 30 21:39:47.152411 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.152379 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm"] Oct 30 21:39:47.169875 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.169835 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzcdf\" (UniqueName: \"kubernetes.io/projected/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-kube-api-access-fzcdf\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.170105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.169973 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-envoy\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.170105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.170014 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-credential-socket\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.170247 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.170176 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-workload-socket\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.170247 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.170240 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-data\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.170349 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.170285 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istiod-ca-cert\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.170396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.170367 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-token\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.170431 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.170397 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-podinfo\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.170478 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.170427 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-workload-certs\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.271413 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.271362 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-token\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.271617 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.271424 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-podinfo\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.271617 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.271455 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-workload-certs\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.271617 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.271555 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzcdf\" (UniqueName: \"kubernetes.io/projected/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-kube-api-access-fzcdf\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.271837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.271629 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-envoy\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.271837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.271662 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-credential-socket\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.271837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.271727 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-workload-socket\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.271837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.271770 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-data\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.271837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.271818 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istiod-ca-cert\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.272276 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.272245 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-workload-socket\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.272387 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.272361 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-credential-socket\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.272470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.272381 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-workload-certs\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.272601 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.272580 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istiod-ca-cert\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.272683 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.272662 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-data\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.274415 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.274394 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-envoy\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.274569 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.274547 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-podinfo\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.292045 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.291992 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-token\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.292190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.292150 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzcdf\" (UniqueName: \"kubernetes.io/projected/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-kube-api-access-fzcdf\") pod \"e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.435706 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.435671 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:47.760164 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.760135 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm"] Oct 30 21:39:47.761539 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:39:47.761502 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239b39fd_6aa2_4f03_8aef_d46a83aafc0c.slice/crio-9f6466b61259abd697caaec350a24ca2f7c35ccf0dc2cef9f14c56a33bd63478 WatchSource:0}: Error finding container 9f6466b61259abd697caaec350a24ca2f7c35ccf0dc2cef9f14c56a33bd63478: Status 404 returned error can't find the container with id 9f6466b61259abd697caaec350a24ca2f7c35ccf0dc2cef9f14c56a33bd63478 Oct 30 21:39:47.764658 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.764617 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:47.764773 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.764716 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:47.764773 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:47.764760 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:39:48.513061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:48.513012 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" event={"ID":"239b39fd-6aa2-4f03-8aef-d46a83aafc0c","Type":"ContainerStarted","Data":"ff11bf4d6d3046adeabfda897c781fe65d4bf5f963245c6188599df871b7c6b0"} Oct 30 21:39:48.513513 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:48.513091 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" event={"ID":"239b39fd-6aa2-4f03-8aef-d46a83aafc0c","Type":"ContainerStarted","Data":"9f6466b61259abd697caaec350a24ca2f7c35ccf0dc2cef9f14c56a33bd63478"} Oct 30 21:39:48.569525 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:48.569427 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" podStartSLOduration=1.569389282 podStartE2EDuration="1.569389282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:48.568352252 +0000 UTC m=+2375.689425998" watchObservedRunningTime="2026-10-30 21:39:48.569389282 +0000 UTC m=+2375.690463029" Oct 30 21:39:49.436153 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:49.436113 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:49.438168 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:49.438138 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 21:39:49.438310 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:49.438205 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" podUID="239b39fd-6aa2-4f03-8aef-d46a83aafc0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 21:39:50.028593 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:50.028552 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c055fc-4bwvq"] Oct 30 21:39:50.039763 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:50.039728 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2c055fc-4bwvq"] Oct 30 21:39:50.437015 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:50.436978 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 21:39:50.437242 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:50.437064 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" podUID="239b39fd-6aa2-4f03-8aef-d46a83aafc0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 21:39:51.019857 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:51.019811 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6f43b83-7049-4398-968a-a2267804b3ed" path="/var/lib/kubelet/pods/d6f43b83-7049-4398-968a-a2267804b3ed/volumes" Oct 30 21:39:51.436762 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:51.436725 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 21:39:51.437160 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:51.436779 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" podUID="239b39fd-6aa2-4f03-8aef-d46a83aafc0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 21:39:52.453777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:52.453742 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 21:39:52.454595 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:52.454555 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" podUID="239b39fd-6aa2-4f03-8aef-d46a83aafc0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 21:39:52.643777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:52.643740 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l"] Oct 30 21:39:52.654250 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:52.654211 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-f23eae-vh57l"] Oct 30 21:39:53.017312 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:53.017269 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d" path="/var/lib/kubelet/pods/95b5d5eb-8e5e-4fcd-b6bc-0aa00c65f28d/volumes" Oct 30 21:39:53.436411 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:53.436372 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 21:39:53.436622 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:53.436429 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" podUID="239b39fd-6aa2-4f03-8aef-d46a83aafc0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 21:39:54.436613 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:54.436572 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 21:39:54.437203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:54.436627 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" podUID="239b39fd-6aa2-4f03-8aef-d46a83aafc0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 21:39:55.442062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:55.441995 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:55.442510 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:55.442488 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:39:55.443254 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:39:55.443234 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:40:00.010855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:00.010816 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:40:04.031723 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:04.031685 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz"] Oct 30 21:40:04.044849 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:04.044813 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3667b84c-bnzzz"] Oct 30 21:40:05.017635 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:05.017604 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b11ccb12-e941-4c28-b18a-7aa50595d7a7" path="/var/lib/kubelet/pods/b11ccb12-e941-4c28-b18a-7aa50595d7a7/volumes" Oct 30 21:40:07.492867 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:40:07.492823 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:40:11.818099 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:11.817984 4926 generic.go:356] "Generic (PLEG): container finished" podID="7c6202f3-5168-4b2a-a167-c87f1fa98fd1" containerID="02609b4cdc3a2acc0da054a95129303a39cd410a1141408f1ac4fc04b737f07e" exitCode=0 Oct 30 21:40:11.818099 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:11.818062 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" event={"ID":"7c6202f3-5168-4b2a-a167-c87f1fa98fd1","Type":"ContainerDied","Data":"02609b4cdc3a2acc0da054a95129303a39cd410a1141408f1ac4fc04b737f07e"} Oct 30 21:40:11.818680 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:11.818582 4926 scope.go:122] "RemoveContainer" containerID="02609b4cdc3a2acc0da054a95129303a39cd410a1141408f1ac4fc04b737f07e" Oct 30 21:40:11.820540 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:11.820508 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:11.820640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:11.820583 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:11.820640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:11.820622 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:12.829255 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:12.829208 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" event={"ID":"7c6202f3-5168-4b2a-a167-c87f1fa98fd1","Type":"ContainerStarted","Data":"0d914076ebb52e1a0d9e902bdead09db6e43f9b9f855dca5b8a131d1584f6026"} Oct 30 21:40:13.361840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:13.361796 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:40:13.560913 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:13.560879 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:40:13.562653 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:13.562624 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:40:13.803418 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:13.803378 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:40:13.805155 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:13.805118 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 21:40:13.805330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:13.805219 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" podUID="7c6202f3-5168-4b2a-a167-c87f1fa98fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 21:40:14.804609 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:14.804547 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 21:40:14.806080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:14.805473 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" podUID="7c6202f3-5168-4b2a-a167-c87f1fa98fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 21:40:15.803503 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:15.803455 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 21:40:15.803739 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:15.803522 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" podUID="7c6202f3-5168-4b2a-a167-c87f1fa98fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 21:40:16.803566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:16.803518 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 21:40:16.804095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:16.803590 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" podUID="7c6202f3-5168-4b2a-a167-c87f1fa98fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 21:40:17.489162 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:17.489124 4926 scope.go:122] "RemoveContainer" containerID="dd13ff620306b31da7c7d441028ac50c999f6648c1ce7080429830870f6ee0a5" Oct 30 21:40:17.522900 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:17.522875 4926 scope.go:122] "RemoveContainer" containerID="340afed749be1b530cf249773db339eb8f85c2a9728aafefb3a670a699ff8b3c" Oct 30 21:40:17.803613 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:17.803530 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" start-of-body= Oct 30 21:40:17.804147 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:17.804109 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" podUID="7c6202f3-5168-4b2a-a167-c87f1fa98fd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.210:15021/healthz/ready\": dial tcp 10.132.0.210:15021: connect: connection refused" Oct 30 21:40:18.808195 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:18.808159 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:40:18.809410 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:18.809382 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:40:27.027630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:27.027589 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0100b84a-8rg8m"] Oct 30 21:40:27.031585 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:27.031559 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0100b84a-8rg8m"] Oct 30 21:40:28.026836 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:28.026797 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec2e75d1-rdc6n"] Oct 30 21:40:28.037189 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:28.037151 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec2e75d1-rdc6n"] Oct 30 21:40:29.026939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:29.026893 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34af379a-15f2-4d6f-a438-f004b9cccb2e" path="/var/lib/kubelet/pods/34af379a-15f2-4d6f-a438-f004b9cccb2e/volumes" Oct 30 21:40:29.027432 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:29.027416 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56d39bbb-f292-4757-93aa-03fa94705e39" path="/var/lib/kubelet/pods/56d39bbb-f292-4757-93aa-03fa94705e39/volumes" Oct 30 21:40:37.486346 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:40:37.486301 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:40:58.316654 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:58.316616 4926 generic.go:356] "Generic (PLEG): container finished" podID="44fcef78-1e16-4d54-9306-52f006c0bb9f" containerID="078e4011583f679da1086c5dbc33fd6e4e37f7bd60d39bc2f56dd5e2ed56107c" exitCode=0 Oct 30 21:40:58.317152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:58.316690 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" event={"ID":"44fcef78-1e16-4d54-9306-52f006c0bb9f","Type":"ContainerDied","Data":"078e4011583f679da1086c5dbc33fd6e4e37f7bd60d39bc2f56dd5e2ed56107c"} Oct 30 21:40:58.317211 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:58.317191 4926 scope.go:122] "RemoveContainer" containerID="078e4011583f679da1086c5dbc33fd6e4e37f7bd60d39bc2f56dd5e2ed56107c" Oct 30 21:40:58.318993 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:58.318961 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:58.319122 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:58.319053 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:58.319122 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:58.319092 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:40:59.326702 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:40:59.326659 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" event={"ID":"44fcef78-1e16-4d54-9306-52f006c0bb9f","Type":"ContainerStarted","Data":"0e29aabd94a498c95d50dd7e7fc44bb7fa88cf6d648357afaf19538380a24582"} Oct 30 21:41:00.029570 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:00.029528 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n"] Oct 30 21:41:00.041018 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:00.040978 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf8f82a0-pcz5n"] Oct 30 21:41:00.311842 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:00.311750 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:41:00.312017 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:00.311994 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 21:41:00.312094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:00.312059 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" podUID="44fcef78-1e16-4d54-9306-52f006c0bb9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 21:41:01.021511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:01.021470 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a213e915-71a7-4c50-b0d4-58f1f5210028" path="/var/lib/kubelet/pods/a213e915-71a7-4c50-b0d4-58f1f5210028/volumes" Oct 30 21:41:01.311305 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:01.311221 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 21:41:01.311305 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:01.311281 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" podUID="44fcef78-1e16-4d54-9306-52f006c0bb9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 21:41:02.311501 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:02.311454 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 21:41:02.312025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:02.311529 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" podUID="44fcef78-1e16-4d54-9306-52f006c0bb9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 21:41:03.027167 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:03.027128 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds"] Oct 30 21:41:03.038288 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:03.038248 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ac19acb-ng7ds"] Oct 30 21:41:03.311589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:03.311492 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 21:41:03.312209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:03.312169 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" podUID="44fcef78-1e16-4d54-9306-52f006c0bb9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 21:41:04.311766 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:04.311722 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Oct 30 21:41:04.312335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:04.311794 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" podUID="44fcef78-1e16-4d54-9306-52f006c0bb9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Oct 30 21:41:05.017431 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:05.017394 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd8d58bc-798f-4e13-ad51-8eba733fb434" path="/var/lib/kubelet/pods/cd8d58bc-798f-4e13-ad51-8eba733fb434/volumes" Oct 30 21:41:05.330840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:05.330736 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:41:05.331605 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:05.331573 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:41:05.332417 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:05.332394 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:41:07.486874 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:41:07.486830 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:41:11.811782 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:11.811736 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf"] Oct 30 21:41:11.816824 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:11.816793 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:11.831508 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:11.831479 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-da52fd-openshift-default-dockercfg-25lhb\"" Oct 30 21:41:11.847439 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:11.847408 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf"] Oct 30 21:41:11.920627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:11.920585 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq6xm\" (UniqueName: \"kubernetes.io/projected/f40cfadc-67d6-467e-bced-347b09dd6462-kube-api-access-mq6xm\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:11.920627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:11.920638 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40cfadc-67d6-467e-bced-347b09dd6462-istio-token\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:11.920926 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:11.920776 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-workload-socket\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:11.920926 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:11.920830 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-credential-socket\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:11.920926 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:11.920861 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-istio-envoy\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:11.921135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:11.920942 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-istio-data\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:11.921135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:11.921052 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-workload-certs\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:11.921135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:11.921107 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40cfadc-67d6-467e-bced-347b09dd6462-istiod-ca-cert\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:11.921302 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:11.921150 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40cfadc-67d6-467e-bced-347b09dd6462-istio-podinfo\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:12.025024 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.024970 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-workload-certs\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:12.025943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.025918 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40cfadc-67d6-467e-bced-347b09dd6462-istiod-ca-cert\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:12.026230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.026197 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40cfadc-67d6-467e-bced-347b09dd6462-istio-podinfo\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:12.026443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.026421 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq6xm\" (UniqueName: \"kubernetes.io/projected/f40cfadc-67d6-467e-bced-347b09dd6462-kube-api-access-mq6xm\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:12.026612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.026594 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40cfadc-67d6-467e-bced-347b09dd6462-istio-token\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:12.026782 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.026763 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-workload-socket\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:12.026938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.026920 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-credential-socket\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:12.027081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.027063 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-istio-envoy\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:12.027268 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.027249 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-istio-data\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:12.042054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.027804 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-istio-data\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:12.042312 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.025846 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-workload-certs\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:12.042448 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.033168 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40cfadc-67d6-467e-bced-347b09dd6462-istiod-ca-cert\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:12.042448 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.033836 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-workload-socket\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:12.042665 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.034369 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-credential-socket\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:12.042665 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.041684 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-istio-envoy\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:12.053231 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.051724 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq6xm\" (UniqueName: \"kubernetes.io/projected/f40cfadc-67d6-467e-bced-347b09dd6462-kube-api-access-mq6xm\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:12.070124 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.069982 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40cfadc-67d6-467e-bced-347b09dd6462-istio-token\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:12.078802 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.078593 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40cfadc-67d6-467e-bced-347b09dd6462-istio-podinfo\") pod \"e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:12.145167 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.145087 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:12.491874 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.491838 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf"] Oct 30 21:41:12.492222 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:41:12.492195 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40cfadc_67d6_467e_bced_347b09dd6462.slice/crio-42257197effc39e67d0f04332cc18aecb7e8a56c59ebd6ce0314615ad189cd0e WatchSource:0}: Error finding container 42257197effc39e67d0f04332cc18aecb7e8a56c59ebd6ce0314615ad189cd0e: Status 404 returned error can't find the container with id 42257197effc39e67d0f04332cc18aecb7e8a56c59ebd6ce0314615ad189cd0e Oct 30 21:41:12.495219 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.495189 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:12.495301 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.495258 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:12.495301 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:12.495287 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:41:13.446017 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:13.445972 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" event={"ID":"f40cfadc-67d6-467e-bced-347b09dd6462","Type":"ContainerStarted","Data":"22d8b63f2e3fe08d22a83b30d53fe671845ff89e68f9a4d6f0cc1ca58206960b"} Oct 30 21:41:13.446017 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:13.446047 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" event={"ID":"f40cfadc-67d6-467e-bced-347b09dd6462","Type":"ContainerStarted","Data":"42257197effc39e67d0f04332cc18aecb7e8a56c59ebd6ce0314615ad189cd0e"} Oct 30 21:41:13.512983 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:13.512927 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" podStartSLOduration=2.512910599 podStartE2EDuration="2.512910599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:13.49844823 +0000 UTC m=+2460.619521977" watchObservedRunningTime="2026-10-30 21:41:13.512910599 +0000 UTC m=+2460.633984345" Oct 30 21:41:14.146016 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:14.145976 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:14.153686 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:14.153652 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:14.454758 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:14.454718 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:14.455859 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:14.455833 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:41:17.571048 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:17.571003 4926 scope.go:122] "RemoveContainer" containerID="da566c0baf4b966a9b55a07fd551b6e3a0a80580c4361acd3fd4387bac929556" Oct 30 21:41:17.585391 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:17.585367 4926 scope.go:122] "RemoveContainer" containerID="5e7a2bb96ec68de0284f31e51dcbc6a505b00f319c12492d30c4dc1481ebbcbb" Oct 30 21:41:18.851302 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:18.851265 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl"] Oct 30 21:41:18.857495 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:18.857462 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:18.879593 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:18.879561 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl"] Oct 30 21:41:18.888174 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:18.888140 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-66cc54\"/\"default-dockercfg-fsbr6\"" Oct 30 21:41:18.890773 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:18.890729 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-66cc54\"/\"openshift-service-ca.crt\"" Oct 30 21:41:18.891196 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:18.891170 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-66cc54\"/\"auth-test-model-66cc54-kserve-self-signed-certs\"" Oct 30 21:41:18.891490 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:18.891435 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-66cc54\"/\"kube-root-ca.crt\"" Oct 30 21:41:18.938666 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:18.938622 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f11288b-8b0d-4e07-8d5d-613780c8f390-tls-certs\") pod \"auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:18.938869 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:18.938765 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfbmn\" (UniqueName: \"kubernetes.io/projected/5f11288b-8b0d-4e07-8d5d-613780c8f390-kube-api-access-vfbmn\") pod \"auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:18.938869 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:18.938824 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-tmp-dir\") pod \"auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:18.939010 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:18.938915 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-dshm\") pod \"auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:18.939089 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:18.939061 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-model-cache\") pod \"auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:18.939151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:18.939132 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-home\") pod \"auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:19.040322 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:19.040281 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f11288b-8b0d-4e07-8d5d-613780c8f390-tls-certs\") pod \"auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:19.040557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:19.040338 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfbmn\" (UniqueName: \"kubernetes.io/projected/5f11288b-8b0d-4e07-8d5d-613780c8f390-kube-api-access-vfbmn\") pod \"auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:19.040557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:19.040362 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-tmp-dir\") pod \"auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:19.040557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:19.040407 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-dshm\") pod \"auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:19.040557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:19.040510 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-model-cache\") pod \"auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:19.040809 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:19.040562 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-home\") pod \"auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:19.040877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:19.040851 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-tmp-dir\") pod \"auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:19.040938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:19.040917 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-model-cache\") pod \"auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:19.041093 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:19.041075 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-home\") pod \"auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:19.042883 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:19.042855 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-dshm\") pod \"auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:19.043424 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:19.043405 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f11288b-8b0d-4e07-8d5d-613780c8f390-tls-certs\") pod \"auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:19.060166 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:19.060131 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfbmn\" (UniqueName: \"kubernetes.io/projected/5f11288b-8b0d-4e07-8d5d-613780c8f390-kube-api-access-vfbmn\") pod \"auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:19.181055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:19.173355 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:19.585148 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:19.582969 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl"] Oct 30 21:41:19.593401 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:41:19.593365 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f11288b_8b0d_4e07_8d5d_613780c8f390.slice/crio-8f542dd1cecef6ced857ae9c7eb3b1c729db6a6ef601c8cd52a1497841c1e55d WatchSource:0}: Error finding container 8f542dd1cecef6ced857ae9c7eb3b1c729db6a6ef601c8cd52a1497841c1e55d: Status 404 returned error can't find the container with id 8f542dd1cecef6ced857ae9c7eb3b1c729db6a6ef601c8cd52a1497841c1e55d Oct 30 21:41:20.043805 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:20.043766 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n"] Oct 30 21:41:20.054647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:20.054615 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dd5ed0f-s2d8n"] Oct 30 21:41:20.627721 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:20.627572 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" event={"ID":"5f11288b-8b0d-4e07-8d5d-613780c8f390","Type":"ContainerStarted","Data":"ba422981ad55bed82859bce250d699ab3b3be47a571f552515c271f8aaec20c1"} Oct 30 21:41:20.627721 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:20.627650 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" event={"ID":"5f11288b-8b0d-4e07-8d5d-613780c8f390","Type":"ContainerStarted","Data":"8f542dd1cecef6ced857ae9c7eb3b1c729db6a6ef601c8cd52a1497841c1e55d"} Oct 30 21:41:20.636129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:20.636092 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:41:20.676621 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:20.676564 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" podStartSLOduration=2.676548041 podStartE2EDuration="2.676548041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:20.673106683 +0000 UTC m=+2467.794180426" watchObservedRunningTime="2026-10-30 21:41:20.676548041 +0000 UTC m=+2467.797621787" Oct 30 21:41:21.020320 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:21.020280 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04df2344-f9e9-4121-887e-0915c6714f8c" path="/var/lib/kubelet/pods/04df2344-f9e9-4121-887e-0915c6714f8c/volumes" Oct 30 21:41:21.174169 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:21.174134 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:25.055890 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:25.055796 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk"] Oct 30 21:41:25.061337 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:25.061310 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk" Oct 30 21:41:25.080923 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:25.080874 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk"] Oct 30 21:41:25.086416 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:25.086347 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:41:25.086617 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:25.086588 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:41:25.157761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:25.157722 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfdkm\" (UniqueName: \"kubernetes.io/projected/22c6c8db-3593-46bc-a7c8-6475696ef182-kube-api-access-wfdkm\") pod \"maas-api-revoke-sub-e8b763b4-mjjhk\" (UID: \"22c6c8db-3593-46bc-a7c8-6475696ef182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk" Oct 30 21:41:25.157969 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:25.157888 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c6c8db-3593-46bc-a7c8-6475696ef182-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8b763b4-mjjhk\" (UID: \"22c6c8db-3593-46bc-a7c8-6475696ef182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk" Oct 30 21:41:25.157969 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:25.157939 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c6c8db-3593-46bc-a7c8-6475696ef182-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8b763b4-mjjhk\" (UID: \"22c6c8db-3593-46bc-a7c8-6475696ef182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk" Oct 30 21:41:25.259078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:25.259044 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfdkm\" (UniqueName: \"kubernetes.io/projected/22c6c8db-3593-46bc-a7c8-6475696ef182-kube-api-access-wfdkm\") pod \"maas-api-revoke-sub-e8b763b4-mjjhk\" (UID: \"22c6c8db-3593-46bc-a7c8-6475696ef182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk" Oct 30 21:41:25.259292 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:25.259239 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c6c8db-3593-46bc-a7c8-6475696ef182-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8b763b4-mjjhk\" (UID: \"22c6c8db-3593-46bc-a7c8-6475696ef182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk" Oct 30 21:41:25.259363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:25.259322 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c6c8db-3593-46bc-a7c8-6475696ef182-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8b763b4-mjjhk\" (UID: \"22c6c8db-3593-46bc-a7c8-6475696ef182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk" Oct 30 21:41:25.260093 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:25.260017 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c6c8db-3593-46bc-a7c8-6475696ef182-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8b763b4-mjjhk\" (UID: \"22c6c8db-3593-46bc-a7c8-6475696ef182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk" Oct 30 21:41:25.261767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:25.261742 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c6c8db-3593-46bc-a7c8-6475696ef182-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8b763b4-mjjhk\" (UID: \"22c6c8db-3593-46bc-a7c8-6475696ef182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk" Oct 30 21:41:25.278342 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:25.278318 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfdkm\" (UniqueName: \"kubernetes.io/projected/22c6c8db-3593-46bc-a7c8-6475696ef182-kube-api-access-wfdkm\") pod \"maas-api-revoke-sub-e8b763b4-mjjhk\" (UID: \"22c6c8db-3593-46bc-a7c8-6475696ef182\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk" Oct 30 21:41:25.379505 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:25.379419 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk" Oct 30 21:41:25.694881 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:25.694843 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk"] Oct 30 21:41:25.704159 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:41:25.704131 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22c6c8db_3593_46bc_a7c8_6475696ef182.slice/crio-87ffc181402e8c76d2add1e0f6be2f94f3d070fc599bee4955fd28d3fb02cbab WatchSource:0}: Error finding container 87ffc181402e8c76d2add1e0f6be2f94f3d070fc599bee4955fd28d3fb02cbab: Status 404 returned error can't find the container with id 87ffc181402e8c76d2add1e0f6be2f94f3d070fc599bee4955fd28d3fb02cbab Oct 30 21:41:26.688187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:26.688139 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk" event={"ID":"22c6c8db-3593-46bc-a7c8-6475696ef182","Type":"ContainerStarted","Data":"912971045adfcaea085c5a1a8d98d3941f82ede5f65d19ed5e96128190af326d"} Oct 30 21:41:26.688743 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:26.688211 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk" event={"ID":"22c6c8db-3593-46bc-a7c8-6475696ef182","Type":"ContainerStarted","Data":"87ffc181402e8c76d2add1e0f6be2f94f3d070fc599bee4955fd28d3fb02cbab"} Oct 30 21:41:26.735381 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:26.735314 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk" podStartSLOduration=1.735297892 podStartE2EDuration="1.735297892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:26.732140978 +0000 UTC m=+2473.853214724" watchObservedRunningTime="2026-10-30 21:41:26.735297892 +0000 UTC m=+2473.856371637" Oct 30 21:41:27.701260 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:27.701221 4926 generic.go:356] "Generic (PLEG): container finished" podID="22c6c8db-3593-46bc-a7c8-6475696ef182" containerID="912971045adfcaea085c5a1a8d98d3941f82ede5f65d19ed5e96128190af326d" exitCode=0 Oct 30 21:41:27.701766 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:27.701306 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk" event={"ID":"22c6c8db-3593-46bc-a7c8-6475696ef182","Type":"ContainerDied","Data":"912971045adfcaea085c5a1a8d98d3941f82ede5f65d19ed5e96128190af326d"} Oct 30 21:41:28.940775 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:28.940742 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk" Oct 30 21:41:29.051250 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:29.051159 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22c6c8db-3593-46bc-a7c8-6475696ef182-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c6c8db-3593-46bc-a7c8-6475696ef182-maas-api-openshift-ca\") pod \"22c6c8db-3593-46bc-a7c8-6475696ef182\" (UID: \"22c6c8db-3593-46bc-a7c8-6475696ef182\") " Oct 30 21:41:29.051409 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:29.051259 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22c6c8db-3593-46bc-a7c8-6475696ef182-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c6c8db-3593-46bc-a7c8-6475696ef182-maas-api-xks-ca\") pod \"22c6c8db-3593-46bc-a7c8-6475696ef182\" (UID: \"22c6c8db-3593-46bc-a7c8-6475696ef182\") " Oct 30 21:41:29.051469 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:29.051430 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22c6c8db-3593-46bc-a7c8-6475696ef182-kube-api-access-wfdkm\" (UniqueName: \"kubernetes.io/projected/22c6c8db-3593-46bc-a7c8-6475696ef182-kube-api-access-wfdkm\") pod \"22c6c8db-3593-46bc-a7c8-6475696ef182\" (UID: \"22c6c8db-3593-46bc-a7c8-6475696ef182\") " Oct 30 21:41:29.051609 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:29.051575 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22c6c8db-3593-46bc-a7c8-6475696ef182-maas-api-openshift-ca" pod "22c6c8db-3593-46bc-a7c8-6475696ef182" (UID: "22c6c8db-3593-46bc-a7c8-6475696ef182"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:29.051815 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:29.051789 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22c6c8db-3593-46bc-a7c8-6475696ef182-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:41:29.053598 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:29.053567 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22c6c8db-3593-46bc-a7c8-6475696ef182-maas-api-xks-ca" pod "22c6c8db-3593-46bc-a7c8-6475696ef182" (UID: "22c6c8db-3593-46bc-a7c8-6475696ef182"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:29.053598 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:29.053583 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22c6c8db-3593-46bc-a7c8-6475696ef182-kube-api-access-wfdkm" pod "22c6c8db-3593-46bc-a7c8-6475696ef182" (UID: "22c6c8db-3593-46bc-a7c8-6475696ef182"). InnerVolumeSpecName "kube-api-access-wfdkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:29.153406 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:29.153360 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfdkm\" (UniqueName: \"kubernetes.io/projected/22c6c8db-3593-46bc-a7c8-6475696ef182-kube-api-access-wfdkm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:41:29.153406 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:29.153405 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22c6c8db-3593-46bc-a7c8-6475696ef182-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:41:29.174366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:29.174324 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:29.187657 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:29.187618 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:29.724011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:29.723974 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk" Oct 30 21:41:29.724011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:29.723992 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk" event={"ID":"22c6c8db-3593-46bc-a7c8-6475696ef182","Type":"ContainerDied","Data":"87ffc181402e8c76d2add1e0f6be2f94f3d070fc599bee4955fd28d3fb02cbab"} Oct 30 21:41:29.724400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:29.724102 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87ffc181402e8c76d2add1e0f6be2f94f3d070fc599bee4955fd28d3fb02cbab" Oct 30 21:41:29.736582 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:29.736550 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:41:35.475666 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:35.475619 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6"] Oct 30 21:41:35.476566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:35.476542 4926 cpu_manager.go:414] "containerMap: removing container" podUID="22c6c8db-3593-46bc-a7c8-6475696ef182" containerName="revoke-keys" Oct 30 21:41:35.476626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:35.476569 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22c6c8db-3593-46bc-a7c8-6475696ef182" containerName="revoke-keys" Oct 30 21:41:35.476738 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:35.476725 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="22c6c8db-3593-46bc-a7c8-6475696ef182" containerName="revoke-keys" Oct 30 21:41:35.481290 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:35.481268 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6" Oct 30 21:41:35.497892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:35.497859 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6"] Oct 30 21:41:35.503273 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:35.503225 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:41:35.503462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:35.503325 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:41:35.640645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:35.640601 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6b9efd-fc70-4387-9608-0a5dbbcc0a56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bf0b5f0-srgl6\" (UID: \"3e6b9efd-fc70-4387-9608-0a5dbbcc0a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6" Oct 30 21:41:35.640852 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:35.640668 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm6kn\" (UniqueName: \"kubernetes.io/projected/3e6b9efd-fc70-4387-9608-0a5dbbcc0a56-kube-api-access-wm6kn\") pod \"maas-api-revoke-sub-7bf0b5f0-srgl6\" (UID: \"3e6b9efd-fc70-4387-9608-0a5dbbcc0a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6" Oct 30 21:41:35.640852 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:35.640812 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6b9efd-fc70-4387-9608-0a5dbbcc0a56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bf0b5f0-srgl6\" (UID: \"3e6b9efd-fc70-4387-9608-0a5dbbcc0a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6" Oct 30 21:41:35.742655 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:35.742552 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6b9efd-fc70-4387-9608-0a5dbbcc0a56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bf0b5f0-srgl6\" (UID: \"3e6b9efd-fc70-4387-9608-0a5dbbcc0a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6" Oct 30 21:41:35.742833 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:35.742793 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6b9efd-fc70-4387-9608-0a5dbbcc0a56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bf0b5f0-srgl6\" (UID: \"3e6b9efd-fc70-4387-9608-0a5dbbcc0a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6" Oct 30 21:41:35.742919 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:35.742901 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm6kn\" (UniqueName: \"kubernetes.io/projected/3e6b9efd-fc70-4387-9608-0a5dbbcc0a56-kube-api-access-wm6kn\") pod \"maas-api-revoke-sub-7bf0b5f0-srgl6\" (UID: \"3e6b9efd-fc70-4387-9608-0a5dbbcc0a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6" Oct 30 21:41:35.743415 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:35.743380 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6b9efd-fc70-4387-9608-0a5dbbcc0a56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7bf0b5f0-srgl6\" (UID: \"3e6b9efd-fc70-4387-9608-0a5dbbcc0a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6" Oct 30 21:41:35.745295 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:35.745269 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6b9efd-fc70-4387-9608-0a5dbbcc0a56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7bf0b5f0-srgl6\" (UID: \"3e6b9efd-fc70-4387-9608-0a5dbbcc0a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6" Oct 30 21:41:35.763105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:35.763075 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm6kn\" (UniqueName: \"kubernetes.io/projected/3e6b9efd-fc70-4387-9608-0a5dbbcc0a56-kube-api-access-wm6kn\") pod \"maas-api-revoke-sub-7bf0b5f0-srgl6\" (UID: \"3e6b9efd-fc70-4387-9608-0a5dbbcc0a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6" Oct 30 21:41:35.799742 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:35.799696 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6" Oct 30 21:41:36.108717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:36.108672 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6"] Oct 30 21:41:36.108887 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:41:36.108863 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6b9efd_fc70_4387_9608_0a5dbbcc0a56.slice/crio-40fbbc59ceade6bb407dc20da05a232c997481cf55a94be6d5cf252b6a1c3766 WatchSource:0}: Error finding container 40fbbc59ceade6bb407dc20da05a232c997481cf55a94be6d5cf252b6a1c3766: Status 404 returned error can't find the container with id 40fbbc59ceade6bb407dc20da05a232c997481cf55a94be6d5cf252b6a1c3766 Oct 30 21:41:36.804546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:36.804423 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6" event={"ID":"3e6b9efd-fc70-4387-9608-0a5dbbcc0a56","Type":"ContainerStarted","Data":"9579c782b3ea91012b61c80778557e3e08148e50e5daac7bd78afbb498bc11e4"} Oct 30 21:41:36.804546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:36.804510 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6" event={"ID":"3e6b9efd-fc70-4387-9608-0a5dbbcc0a56","Type":"ContainerStarted","Data":"40fbbc59ceade6bb407dc20da05a232c997481cf55a94be6d5cf252b6a1c3766"} Oct 30 21:41:36.842169 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:36.842108 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6" podStartSLOduration=1.8420922910000002 podStartE2EDuration="1.842092291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:36.841411017 +0000 UTC m=+2483.962484764" watchObservedRunningTime="2026-10-30 21:41:36.842092291 +0000 UTC m=+2483.963166033" Oct 30 21:41:37.490119 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:41:37.490070 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:41:37.815004 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:37.814912 4926 generic.go:356] "Generic (PLEG): container finished" podID="3e6b9efd-fc70-4387-9608-0a5dbbcc0a56" containerID="9579c782b3ea91012b61c80778557e3e08148e50e5daac7bd78afbb498bc11e4" exitCode=0 Oct 30 21:41:37.815004 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:37.814965 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6" event={"ID":"3e6b9efd-fc70-4387-9608-0a5dbbcc0a56","Type":"ContainerDied","Data":"9579c782b3ea91012b61c80778557e3e08148e50e5daac7bd78afbb498bc11e4"} Oct 30 21:41:38.999297 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:38.999266 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6" Oct 30 21:41:39.107550 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:39.107433 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e6b9efd-fc70-4387-9608-0a5dbbcc0a56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6b9efd-fc70-4387-9608-0a5dbbcc0a56-maas-api-openshift-ca\") pod \"3e6b9efd-fc70-4387-9608-0a5dbbcc0a56\" (UID: \"3e6b9efd-fc70-4387-9608-0a5dbbcc0a56\") " Oct 30 21:41:39.107757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:39.107583 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e6b9efd-fc70-4387-9608-0a5dbbcc0a56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6b9efd-fc70-4387-9608-0a5dbbcc0a56-maas-api-xks-ca\") pod \"3e6b9efd-fc70-4387-9608-0a5dbbcc0a56\" (UID: \"3e6b9efd-fc70-4387-9608-0a5dbbcc0a56\") " Oct 30 21:41:39.107757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:39.107738 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6b9efd-fc70-4387-9608-0a5dbbcc0a56-kube-api-access-wm6kn\" (UniqueName: \"kubernetes.io/projected/3e6b9efd-fc70-4387-9608-0a5dbbcc0a56-kube-api-access-wm6kn\") pod \"3e6b9efd-fc70-4387-9608-0a5dbbcc0a56\" (UID: \"3e6b9efd-fc70-4387-9608-0a5dbbcc0a56\") " Oct 30 21:41:39.107897 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:39.107863 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e6b9efd-fc70-4387-9608-0a5dbbcc0a56-maas-api-openshift-ca" pod "3e6b9efd-fc70-4387-9608-0a5dbbcc0a56" (UID: "3e6b9efd-fc70-4387-9608-0a5dbbcc0a56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:41:39.108968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:39.108944 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6b9efd-fc70-4387-9608-0a5dbbcc0a56-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:41:39.110106 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:39.110073 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e6b9efd-fc70-4387-9608-0a5dbbcc0a56-maas-api-xks-ca" pod "3e6b9efd-fc70-4387-9608-0a5dbbcc0a56" (UID: "3e6b9efd-fc70-4387-9608-0a5dbbcc0a56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:41:39.110429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:39.110400 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6b9efd-fc70-4387-9608-0a5dbbcc0a56-kube-api-access-wm6kn" pod "3e6b9efd-fc70-4387-9608-0a5dbbcc0a56" (UID: "3e6b9efd-fc70-4387-9608-0a5dbbcc0a56"). InnerVolumeSpecName "kube-api-access-wm6kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:41:39.210800 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:39.210746 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm6kn\" (UniqueName: \"kubernetes.io/projected/3e6b9efd-fc70-4387-9608-0a5dbbcc0a56-kube-api-access-wm6kn\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:41:39.210800 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:39.210792 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6b9efd-fc70-4387-9608-0a5dbbcc0a56-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:41:39.867061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:39.859478 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6" event={"ID":"3e6b9efd-fc70-4387-9608-0a5dbbcc0a56","Type":"ContainerDied","Data":"40fbbc59ceade6bb407dc20da05a232c997481cf55a94be6d5cf252b6a1c3766"} Oct 30 21:41:39.867061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:39.859560 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40fbbc59ceade6bb407dc20da05a232c997481cf55a94be6d5cf252b6a1c3766" Oct 30 21:41:39.867061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:39.859650 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6" Oct 30 21:41:46.033784 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:46.033730 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p"] Oct 30 21:41:46.046216 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:46.046179 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b8dbb49-7r99p"] Oct 30 21:41:47.019132 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:47.019093 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbccc867-07a9-4f40-a181-bd0a6e42571e" path="/var/lib/kubelet/pods/bbccc867-07a9-4f40-a181-bd0a6e42571e/volumes" Oct 30 21:41:56.032401 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:56.032358 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa1493c0-4vgv6"] Oct 30 21:41:56.043299 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:56.043260 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa1493c0-4vgv6"] Oct 30 21:41:57.029633 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:41:57.029574 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="482c7875-59c8-4eed-a756-016214ea4127" path="/var/lib/kubelet/pods/482c7875-59c8-4eed-a756-016214ea4127/volumes" Oct 30 21:42:07.488391 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:42:07.488345 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:42:11.030357 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:11.030308 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc"] Oct 30 21:42:11.041400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:11.041358 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4cd1c85-s5xvc"] Oct 30 21:42:13.020504 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:13.020469 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b496030-08c9-4bec-9ded-8d9b40f2f66b" path="/var/lib/kubelet/pods/3b496030-08c9-4bec-9ded-8d9b40f2f66b/volumes" Oct 30 21:42:17.630493 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:17.630448 4926 scope.go:122] "RemoveContainer" containerID="bfdea7644e6c41ac2645bad7e6635397ef8b6bc22af4148c03a8075eca18c395" Oct 30 21:42:17.645341 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:17.645316 4926 scope.go:122] "RemoveContainer" containerID="e403b0a0d74482ccd0c2a88833b2db093aeda11dece44a97626c506bb92ebe8a" Oct 30 21:42:17.656859 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:17.656835 4926 scope.go:122] "RemoveContainer" containerID="44aa6822f33035b09a249943fb708d9f53330b0684c7f0ad02a2ff2b6e688c72" Oct 30 21:42:17.698619 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:17.696232 4926 scope.go:122] "RemoveContainer" containerID="f49becfd602a9db4615b1084eb4e390ac55e87a33a5ca7ce5397258fea07955e" Oct 30 21:42:17.723301 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:17.723275 4926 scope.go:122] "RemoveContainer" containerID="258dc55b949406425aa1582e498a0708577e5c6042633a97c5e51ea98a46b309" Oct 30 21:42:17.736063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:17.736012 4926 scope.go:122] "RemoveContainer" containerID="2709840fba0af87eeb0f4f2236e097896ad662ac1cdc2f23c3f90ee5d94d3d2e" Oct 30 21:42:17.747296 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:17.747266 4926 scope.go:122] "RemoveContainer" containerID="3fb7e33adf28861a7b7db1f913cd3a82a399e72963220774091d06a4b41a3fe7" Oct 30 21:42:17.760795 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:17.760767 4926 scope.go:122] "RemoveContainer" containerID="e5d179fac78e48c1dbf86dc4196f7935b664e23e4947b84f5c5fb7669b13e537" Oct 30 21:42:23.328527 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:23.328489 4926 generic.go:356] "Generic (PLEG): container finished" podID="f40cfadc-67d6-467e-bced-347b09dd6462" containerID="22d8b63f2e3fe08d22a83b30d53fe671845ff89e68f9a4d6f0cc1ca58206960b" exitCode=0 Oct 30 21:42:23.329400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:23.328566 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" event={"ID":"f40cfadc-67d6-467e-bced-347b09dd6462","Type":"ContainerDied","Data":"22d8b63f2e3fe08d22a83b30d53fe671845ff89e68f9a4d6f0cc1ca58206960b"} Oct 30 21:42:23.329400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:23.329089 4926 scope.go:122] "RemoveContainer" containerID="22d8b63f2e3fe08d22a83b30d53fe671845ff89e68f9a4d6f0cc1ca58206960b" Oct 30 21:42:23.330920 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:23.330889 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:42:23.330995 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:23.330968 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:42:23.331058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:23.331002 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:42:24.338845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:24.338805 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" event={"ID":"f40cfadc-67d6-467e-bced-347b09dd6462","Type":"ContainerStarted","Data":"b5c917de81666e2c397c8dbe044a742efa7b64a2643d69d6548e452235e8fb0b"} Oct 30 21:42:25.146126 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:25.146073 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:42:25.147690 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:25.147659 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 21:42:25.147857 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:25.147730 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" podUID="f40cfadc-67d6-467e-bced-347b09dd6462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 21:42:26.145632 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:26.145588 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 21:42:26.146214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:26.145659 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" podUID="f40cfadc-67d6-467e-bced-347b09dd6462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 21:42:27.146001 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:27.145953 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 21:42:27.146553 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:27.146028 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" podUID="f40cfadc-67d6-467e-bced-347b09dd6462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 21:42:28.146359 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:28.146310 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 21:42:28.146807 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:28.146385 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" podUID="f40cfadc-67d6-467e-bced-347b09dd6462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 21:42:29.145957 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:29.145903 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 21:42:29.146180 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:29.145974 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" podUID="f40cfadc-67d6-467e-bced-347b09dd6462" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 21:42:29.455492 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:29.455450 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:42:30.152076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:30.152009 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:42:30.415746 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:30.415714 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:42:37.490221 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:42:37.489060 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:42:44.717057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:44.716582 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb"] Oct 30 21:42:44.717581 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:44.717527 4926 cpu_manager.go:414] "containerMap: removing container" podUID="3e6b9efd-fc70-4387-9608-0a5dbbcc0a56" containerName="revoke-keys" Oct 30 21:42:44.717581 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:44.717550 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6b9efd-fc70-4387-9608-0a5dbbcc0a56" containerName="revoke-keys" Oct 30 21:42:44.736056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:44.717736 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6b9efd-fc70-4387-9608-0a5dbbcc0a56" containerName="revoke-keys" Oct 30 21:42:44.756120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:44.747322 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb" Oct 30 21:42:44.773287 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:44.772530 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb"] Oct 30 21:42:44.780924 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:44.778280 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-cdcf9d-dockercfg-t6qcw\"" Oct 30 21:42:44.780924 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:44.778508 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-cdcf9d\"" Oct 30 21:42:44.838894 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:44.838850 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d63a5ae-aa39-49f9-bac5-b5427e58e7d3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb\" (UID: \"4d63a5ae-aa39-49f9-bac5-b5427e58e7d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb" Oct 30 21:42:44.839121 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:44.839025 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52dt2\" (UniqueName: \"kubernetes.io/projected/4d63a5ae-aa39-49f9-bac5-b5427e58e7d3-kube-api-access-52dt2\") pod \"payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb\" (UID: \"4d63a5ae-aa39-49f9-bac5-b5427e58e7d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb" Oct 30 21:42:44.940741 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:44.940696 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d63a5ae-aa39-49f9-bac5-b5427e58e7d3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb\" (UID: \"4d63a5ae-aa39-49f9-bac5-b5427e58e7d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb" Oct 30 21:42:44.940935 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:44.940843 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52dt2\" (UniqueName: \"kubernetes.io/projected/4d63a5ae-aa39-49f9-bac5-b5427e58e7d3-kube-api-access-52dt2\") pod \"payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb\" (UID: \"4d63a5ae-aa39-49f9-bac5-b5427e58e7d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb" Oct 30 21:42:44.941704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:44.941682 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d63a5ae-aa39-49f9-bac5-b5427e58e7d3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb\" (UID: \"4d63a5ae-aa39-49f9-bac5-b5427e58e7d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb" Oct 30 21:42:44.959626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:44.959596 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52dt2\" (UniqueName: \"kubernetes.io/projected/4d63a5ae-aa39-49f9-bac5-b5427e58e7d3-kube-api-access-52dt2\") pod \"payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb\" (UID: \"4d63a5ae-aa39-49f9-bac5-b5427e58e7d3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb" Oct 30 21:42:45.104930 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:45.104843 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb" Oct 30 21:42:45.409850 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:45.409817 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb"] Oct 30 21:42:45.411706 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:42:45.411670 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d63a5ae_aa39_49f9_bac5_b5427e58e7d3.slice/crio-3bc5ac638c858abb79a8cff8879410dbc7ad744f8fee1a6940c27cfa95ad6030 WatchSource:0}: Error finding container 3bc5ac638c858abb79a8cff8879410dbc7ad744f8fee1a6940c27cfa95ad6030: Status 404 returned error can't find the container with id 3bc5ac638c858abb79a8cff8879410dbc7ad744f8fee1a6940c27cfa95ad6030 Oct 30 21:42:45.553788 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:45.553749 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb" event={"ID":"4d63a5ae-aa39-49f9-bac5-b5427e58e7d3","Type":"ContainerStarted","Data":"44bf2f7dd25660b0af257eb8878a0840a79f5b921aecdb635aeed32ff4cd05f7"} Oct 30 21:42:45.553788 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:45.553805 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb" event={"ID":"4d63a5ae-aa39-49f9-bac5-b5427e58e7d3","Type":"ContainerStarted","Data":"3bc5ac638c858abb79a8cff8879410dbc7ad744f8fee1a6940c27cfa95ad6030"} Oct 30 21:42:45.554100 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:45.553848 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb" Oct 30 21:42:45.593158 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:45.593097 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb" podStartSLOduration=1.593080319 podStartE2EDuration="1.593080319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:45.591003045 +0000 UTC m=+2552.712076791" watchObservedRunningTime="2026-10-30 21:42:45.593080319 +0000 UTC m=+2552.714154065" Oct 30 21:42:56.566461 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:42:56.566378 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb" Oct 30 21:43:07.486847 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:43:07.486797 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:43:17.121157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.121116 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz"] Oct 30 21:43:17.151418 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.151362 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:17.172898 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.172861 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz"] Oct 30 21:43:17.176949 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.176892 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-da52fd\"/\"openshift-service-ca.crt\"" Oct 30 21:43:17.177315 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.177276 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-da52fd\"/\"kube-root-ca.crt\"" Oct 30 21:43:17.178332 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.178221 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-da52fd\"/\"rate-test-model-da52fd-kserve-self-signed-certs\"" Oct 30 21:43:17.178332 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.178303 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-da52fd\"/\"default-dockercfg-p2zd7\"" Oct 30 21:43:17.239555 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.239511 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-model-cache\") pod \"rate-test-model-da52fd-kserve-76fff44757-qwchz\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:17.239774 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.239565 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-dshm\") pod \"rate-test-model-da52fd-kserve-76fff44757-qwchz\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:17.239774 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.239724 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-home\") pod \"rate-test-model-da52fd-kserve-76fff44757-qwchz\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:17.239936 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.239916 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkcmm\" (UniqueName: \"kubernetes.io/projected/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-kube-api-access-bkcmm\") pod \"rate-test-model-da52fd-kserve-76fff44757-qwchz\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:17.240060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.240022 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-tls-certs\") pod \"rate-test-model-da52fd-kserve-76fff44757-qwchz\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:17.240356 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.240333 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-tmp-dir\") pod \"rate-test-model-da52fd-kserve-76fff44757-qwchz\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:17.341572 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.341517 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkcmm\" (UniqueName: \"kubernetes.io/projected/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-kube-api-access-bkcmm\") pod \"rate-test-model-da52fd-kserve-76fff44757-qwchz\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:17.341811 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.341627 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-tls-certs\") pod \"rate-test-model-da52fd-kserve-76fff44757-qwchz\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:17.341811 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.341743 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-tmp-dir\") pod \"rate-test-model-da52fd-kserve-76fff44757-qwchz\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:17.341811 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.341783 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-model-cache\") pod \"rate-test-model-da52fd-kserve-76fff44757-qwchz\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:17.342003 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.341825 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-dshm\") pod \"rate-test-model-da52fd-kserve-76fff44757-qwchz\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:17.342003 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.341872 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-home\") pod \"rate-test-model-da52fd-kserve-76fff44757-qwchz\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:17.342315 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.342281 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-model-cache\") pod \"rate-test-model-da52fd-kserve-76fff44757-qwchz\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:17.342471 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.342340 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-tmp-dir\") pod \"rate-test-model-da52fd-kserve-76fff44757-qwchz\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:17.342471 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.342417 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-home\") pod \"rate-test-model-da52fd-kserve-76fff44757-qwchz\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:17.344242 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.344216 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-dshm\") pod \"rate-test-model-da52fd-kserve-76fff44757-qwchz\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:17.344770 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.344751 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-tls-certs\") pod \"rate-test-model-da52fd-kserve-76fff44757-qwchz\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:17.366015 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.365973 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkcmm\" (UniqueName: \"kubernetes.io/projected/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-kube-api-access-bkcmm\") pod \"rate-test-model-da52fd-kserve-76fff44757-qwchz\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:17.498067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.497403 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:17.870655 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.870559 4926 scope.go:122] "RemoveContainer" containerID="6ed84bda7b7d5397f6823381460eb17f1d4413e8f17d0f798e655ea1a423fd76" Oct 30 21:43:17.898406 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.896021 4926 scope.go:122] "RemoveContainer" containerID="36676f3b63b4a670fa3045cc28889f12dfae32eba71e756b9042a6bfee4b9dfc" Oct 30 21:43:17.925138 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.924807 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz"] Oct 30 21:43:17.928666 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:43:17.928513 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8baf34ec_5ec8_4bb1_b17a_8b6a3f39b80d.slice/crio-4f8b82e6f3501d1b0c14ea4514e9c19ce537934da3e11e105b089a72aa2c367f WatchSource:0}: Error finding container 4f8b82e6f3501d1b0c14ea4514e9c19ce537934da3e11e105b089a72aa2c367f: Status 404 returned error can't find the container with id 4f8b82e6f3501d1b0c14ea4514e9c19ce537934da3e11e105b089a72aa2c367f Oct 30 21:43:17.936653 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.936629 4926 scope.go:122] "RemoveContainer" containerID="d0f39364d744f0c06c0aff5244391ae7d4b86af5d9de25d2cc855a1613a314f6" Oct 30 21:43:17.951200 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.951173 4926 scope.go:122] "RemoveContainer" containerID="fe436d04d360d00adf7a1e2505c286f9282703da97edf38f7242a59c2f5a3b7e" Oct 30 21:43:17.963005 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.962976 4926 scope.go:122] "RemoveContainer" containerID="4d1eeeac674ddaca267e80cbdf1e231d03a2a86dc9d2d45a85730b7e027ce3ec" Oct 30 21:43:17.981653 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:17.981624 4926 scope.go:122] "RemoveContainer" containerID="ebdf2613534404d72ed8510da406901ea6001254ebb63b999b5b697f44d9db7f" Oct 30 21:43:18.821510 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:18.821457 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:43:18.870211 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:18.870171 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" event={"ID":"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d","Type":"ContainerStarted","Data":"089abf3af1fe3d1a8b01b1363f1819c97b4d38efc8a414fa6d48e75b683b139d"} Oct 30 21:43:18.870211 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:18.870225 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" event={"ID":"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d","Type":"ContainerStarted","Data":"4f8b82e6f3501d1b0c14ea4514e9c19ce537934da3e11e105b089a72aa2c367f"} Oct 30 21:43:18.919444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:18.919374 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" podStartSLOduration=1.919353648 podStartE2EDuration="1.919353648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:18.91645934 +0000 UTC m=+2586.037533087" watchObservedRunningTime="2026-10-30 21:43:18.919353648 +0000 UTC m=+2586.040427395" Oct 30 21:43:19.500330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:19.497797 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:20.031593 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:20.031554 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n"] Oct 30 21:43:20.041984 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:20.041944 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07367a00-ctv2n"] Oct 30 21:43:21.016527 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:21.016492 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f30abe34-8aee-4dce-ac89-155e1a0248bc" path="/var/lib/kubelet/pods/f30abe34-8aee-4dce-ac89-155e1a0248bc/volumes" Oct 30 21:43:27.498863 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:27.498820 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:27.512491 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:27.512455 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:27.957180 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:27.957146 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:43:37.487909 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:43:37.487858 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:43:39.040576 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:39.040539 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb"] Oct 30 21:43:39.055938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:39.055903 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfb3b2f3-94gjb"] Oct 30 21:43:41.017326 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:41.017292 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc629bb-3e16-4b37-a39e-8ae97e732e84" path="/var/lib/kubelet/pods/ffc629bb-3e16-4b37-a39e-8ae97e732e84/volumes" Oct 30 21:43:46.515297 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.515254 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7"] Oct 30 21:43:46.519709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.519675 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.541154 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.541122 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-55c5e0-openshift-default-dockercfg-vt8nw\"" Oct 30 21:43:46.556346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.556302 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7"] Oct 30 21:43:46.573446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.573395 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-envoy\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.573767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.573745 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-token\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.573972 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.573955 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-credential-socket\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.574147 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.574131 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-podinfo\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.574393 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.574374 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-workload-certs\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.574751 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.574732 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97575fd5-0609-4000-904d-ad4cf40f23ef-istiod-ca-cert\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.574983 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.574966 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-data\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.575181 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.575162 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnkfv\" (UniqueName: \"kubernetes.io/projected/97575fd5-0609-4000-904d-ad4cf40f23ef-kube-api-access-cnkfv\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.575380 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.575364 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-workload-socket\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.677214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.677172 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-workload-certs\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.677214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.677214 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97575fd5-0609-4000-904d-ad4cf40f23ef-istiod-ca-cert\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.677517 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.677268 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-data\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.677517 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.677290 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnkfv\" (UniqueName: \"kubernetes.io/projected/97575fd5-0609-4000-904d-ad4cf40f23ef-kube-api-access-cnkfv\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.677517 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.677329 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-workload-socket\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.677517 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.677430 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-envoy\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.677517 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.677469 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-token\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.677517 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.677513 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-credential-socket\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.677825 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.677543 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-podinfo\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.677825 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.677692 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-workload-certs\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.677825 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.677770 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-workload-socket\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.678088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.678026 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-credential-socket\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.678227 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.678204 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97575fd5-0609-4000-904d-ad4cf40f23ef-istiod-ca-cert\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.678320 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.678290 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-data\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.680180 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.680155 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-envoy\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.680611 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.680587 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-podinfo\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.704541 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.704506 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-token\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.705595 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.705565 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnkfv\" (UniqueName: \"kubernetes.io/projected/97575fd5-0609-4000-904d-ad4cf40f23ef-kube-api-access-cnkfv\") pod \"e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:46.836633 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:46.836533 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:47.193181 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:47.193140 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7"] Oct 30 21:43:47.194871 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:43:47.194826 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97575fd5_0609_4000_904d_ad4cf40f23ef.slice/crio-e0ec45cf4e9cad6f77d71f2e5ff7f5e1d9913d8f0972df5e54a3feba0a276ed1 WatchSource:0}: Error finding container e0ec45cf4e9cad6f77d71f2e5ff7f5e1d9913d8f0972df5e54a3feba0a276ed1: Status 404 returned error can't find the container with id e0ec45cf4e9cad6f77d71f2e5ff7f5e1d9913d8f0972df5e54a3feba0a276ed1 Oct 30 21:43:47.198074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:47.198008 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:43:47.198198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:47.198104 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:43:47.198198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:47.198135 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:43:48.126817 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:48.126779 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" event={"ID":"97575fd5-0609-4000-904d-ad4cf40f23ef","Type":"ContainerStarted","Data":"4d42041383fe6050cabee4fc7c005f08c26fa95694420868934e1020598e06d1"} Oct 30 21:43:48.127264 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:48.126838 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" event={"ID":"97575fd5-0609-4000-904d-ad4cf40f23ef","Type":"ContainerStarted","Data":"e0ec45cf4e9cad6f77d71f2e5ff7f5e1d9913d8f0972df5e54a3feba0a276ed1"} Oct 30 21:43:48.201482 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:48.201424 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" podStartSLOduration=2.201407624 podStartE2EDuration="2.201407624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:48.196573713 +0000 UTC m=+2615.317647460" watchObservedRunningTime="2026-10-30 21:43:48.201407624 +0000 UTC m=+2615.322481369" Oct 30 21:43:48.837524 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:48.837483 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:48.839259 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:48.839233 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 21:43:48.839406 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:48.839281 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" podUID="97575fd5-0609-4000-904d-ad4cf40f23ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 21:43:49.837163 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:49.837103 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 21:43:49.837635 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:49.837203 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" podUID="97575fd5-0609-4000-904d-ad4cf40f23ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 21:43:50.837165 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:50.837130 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 21:43:50.837594 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:50.837188 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" podUID="97575fd5-0609-4000-904d-ad4cf40f23ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 21:43:51.837779 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:51.837732 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 21:43:51.838337 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:51.837797 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" podUID="97575fd5-0609-4000-904d-ad4cf40f23ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 21:43:52.837599 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:52.837563 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" start-of-body= Oct 30 21:43:52.837828 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:52.837632 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" podUID="97575fd5-0609-4000-904d-ad4cf40f23ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.223:15021/healthz/ready\": dial tcp 10.132.0.223:15021: connect: connection refused" Oct 30 21:43:53.842534 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:53.842498 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:53.843070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:53.843047 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:53.843842 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:53.843822 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:43:59.455861 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:43:59.455814 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb"] Oct 30 21:44:04.456760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:04.456681 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb" podUID="4d63a5ae-aa39-49f9-bac5-b5427e58e7d3" containerName="payload-processing" containerID="cri-o://44bf2f7dd25660b0af257eb8878a0840a79f5b921aecdb635aeed32ff4cd05f7" gracePeriod=25 Oct 30 21:44:04.780177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:04.780146 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb" Oct 30 21:44:04.898072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:04.897989 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d63a5ae-aa39-49f9-bac5-b5427e58e7d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d63a5ae-aa39-49f9-bac5-b5427e58e7d3-plugins-config-volume\") pod \"4d63a5ae-aa39-49f9-bac5-b5427e58e7d3\" (UID: \"4d63a5ae-aa39-49f9-bac5-b5427e58e7d3\") " Oct 30 21:44:04.898260 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:04.898246 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d63a5ae-aa39-49f9-bac5-b5427e58e7d3-kube-api-access-52dt2\" (UniqueName: \"kubernetes.io/projected/4d63a5ae-aa39-49f9-bac5-b5427e58e7d3-kube-api-access-52dt2\") pod \"4d63a5ae-aa39-49f9-bac5-b5427e58e7d3\" (UID: \"4d63a5ae-aa39-49f9-bac5-b5427e58e7d3\") " Oct 30 21:44:04.898482 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:04.898455 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d63a5ae-aa39-49f9-bac5-b5427e58e7d3-plugins-config-volume" pod "4d63a5ae-aa39-49f9-bac5-b5427e58e7d3" (UID: "4d63a5ae-aa39-49f9-bac5-b5427e58e7d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:04.898817 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:04.898796 4926 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d63a5ae-aa39-49f9-bac5-b5427e58e7d3-plugins-config-volume\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:44:04.900601 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:04.900568 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d63a5ae-aa39-49f9-bac5-b5427e58e7d3-kube-api-access-52dt2" pod "4d63a5ae-aa39-49f9-bac5-b5427e58e7d3" (UID: "4d63a5ae-aa39-49f9-bac5-b5427e58e7d3"). InnerVolumeSpecName "kube-api-access-52dt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:04.999754 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:04.999637 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52dt2\" (UniqueName: \"kubernetes.io/projected/4d63a5ae-aa39-49f9-bac5-b5427e58e7d3-kube-api-access-52dt2\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:44:05.289932 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:05.289827 4926 generic.go:356] "Generic (PLEG): container finished" podID="4d63a5ae-aa39-49f9-bac5-b5427e58e7d3" containerID="44bf2f7dd25660b0af257eb8878a0840a79f5b921aecdb635aeed32ff4cd05f7" exitCode=0 Oct 30 21:44:05.289932 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:05.289901 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb" Oct 30 21:44:05.290253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:05.289908 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb" event={"ID":"4d63a5ae-aa39-49f9-bac5-b5427e58e7d3","Type":"ContainerDied","Data":"44bf2f7dd25660b0af257eb8878a0840a79f5b921aecdb635aeed32ff4cd05f7"} Oct 30 21:44:05.290410 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:05.290295 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb" event={"ID":"4d63a5ae-aa39-49f9-bac5-b5427e58e7d3","Type":"ContainerDied","Data":"3bc5ac638c858abb79a8cff8879410dbc7ad744f8fee1a6940c27cfa95ad6030"} Oct 30 21:44:05.290410 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:05.290331 4926 scope.go:122] "RemoveContainer" containerID="44bf2f7dd25660b0af257eb8878a0840a79f5b921aecdb635aeed32ff4cd05f7" Oct 30 21:44:05.302661 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:05.302637 4926 scope.go:122] "RemoveContainer" containerID="44bf2f7dd25660b0af257eb8878a0840a79f5b921aecdb635aeed32ff4cd05f7" Oct 30 21:44:05.303005 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:44:05.302985 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44bf2f7dd25660b0af257eb8878a0840a79f5b921aecdb635aeed32ff4cd05f7\": container with ID starting with 44bf2f7dd25660b0af257eb8878a0840a79f5b921aecdb635aeed32ff4cd05f7 not found: ID does not exist" containerID="44bf2f7dd25660b0af257eb8878a0840a79f5b921aecdb635aeed32ff4cd05f7" Oct 30 21:44:05.303130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:05.303011 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44bf2f7dd25660b0af257eb8878a0840a79f5b921aecdb635aeed32ff4cd05f7"} err="failed to get container status \"44bf2f7dd25660b0af257eb8878a0840a79f5b921aecdb635aeed32ff4cd05f7\": rpc error: code = NotFound desc = could not find container \"44bf2f7dd25660b0af257eb8878a0840a79f5b921aecdb635aeed32ff4cd05f7\": container with ID starting with 44bf2f7dd25660b0af257eb8878a0840a79f5b921aecdb635aeed32ff4cd05f7 not found: ID does not exist" Oct 30 21:44:05.353207 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:05.353171 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb"] Oct 30 21:44:05.361263 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:05.361234 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-cdcf9d-cc578d77d-mdttb"] Oct 30 21:44:07.018829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:07.018798 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d63a5ae-aa39-49f9-bac5-b5427e58e7d3" path="/var/lib/kubelet/pods/4d63a5ae-aa39-49f9-bac5-b5427e58e7d3/volumes" Oct 30 21:44:07.487447 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:44:07.487400 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:44:16.368845 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:16.368807 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9"] Oct 30 21:44:16.369645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:16.369621 4926 cpu_manager.go:414] "containerMap: removing container" podUID="4d63a5ae-aa39-49f9-bac5-b5427e58e7d3" containerName="payload-processing" Oct 30 21:44:16.369645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:16.369645 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d63a5ae-aa39-49f9-bac5-b5427e58e7d3" containerName="payload-processing" Oct 30 21:44:16.369861 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:16.369845 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d63a5ae-aa39-49f9-bac5-b5427e58e7d3" containerName="payload-processing" Oct 30 21:44:16.373389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:16.373366 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9" Oct 30 21:44:16.396242 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:16.396203 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9"] Oct 30 21:44:16.399189 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:16.399164 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:44:16.399554 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:16.399530 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:44:16.415000 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:16.414966 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vfpv\" (UniqueName: \"kubernetes.io/projected/9cfebe73-9386-47d7-9d93-46e0f38b5b71-kube-api-access-5vfpv\") pod \"maas-api-revoke-sub-86dd7365-krmb9\" (UID: \"9cfebe73-9386-47d7-9d93-46e0f38b5b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9" Oct 30 21:44:16.415247 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:16.415071 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cfebe73-9386-47d7-9d93-46e0f38b5b71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86dd7365-krmb9\" (UID: \"9cfebe73-9386-47d7-9d93-46e0f38b5b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9" Oct 30 21:44:16.415247 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:16.415128 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cfebe73-9386-47d7-9d93-46e0f38b5b71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86dd7365-krmb9\" (UID: \"9cfebe73-9386-47d7-9d93-46e0f38b5b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9" Oct 30 21:44:16.516966 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:16.516583 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vfpv\" (UniqueName: \"kubernetes.io/projected/9cfebe73-9386-47d7-9d93-46e0f38b5b71-kube-api-access-5vfpv\") pod \"maas-api-revoke-sub-86dd7365-krmb9\" (UID: \"9cfebe73-9386-47d7-9d93-46e0f38b5b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9" Oct 30 21:44:16.516966 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:16.516694 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cfebe73-9386-47d7-9d93-46e0f38b5b71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86dd7365-krmb9\" (UID: \"9cfebe73-9386-47d7-9d93-46e0f38b5b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9" Oct 30 21:44:16.516966 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:16.516740 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cfebe73-9386-47d7-9d93-46e0f38b5b71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86dd7365-krmb9\" (UID: \"9cfebe73-9386-47d7-9d93-46e0f38b5b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9" Oct 30 21:44:16.517626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:16.517594 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cfebe73-9386-47d7-9d93-46e0f38b5b71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-86dd7365-krmb9\" (UID: \"9cfebe73-9386-47d7-9d93-46e0f38b5b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9" Oct 30 21:44:16.520569 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:16.520403 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cfebe73-9386-47d7-9d93-46e0f38b5b71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-86dd7365-krmb9\" (UID: \"9cfebe73-9386-47d7-9d93-46e0f38b5b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9" Oct 30 21:44:16.542148 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:16.542120 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vfpv\" (UniqueName: \"kubernetes.io/projected/9cfebe73-9386-47d7-9d93-46e0f38b5b71-kube-api-access-5vfpv\") pod \"maas-api-revoke-sub-86dd7365-krmb9\" (UID: \"9cfebe73-9386-47d7-9d93-46e0f38b5b71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9" Oct 30 21:44:16.693226 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:16.693180 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9" Oct 30 21:44:17.019265 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:44:17.019219 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cfebe73_9386_47d7_9d93_46e0f38b5b71.slice/crio-2070a42c5b15ec1f36e93d061ddccc97a8388a50ab7b3277c82fc52179f4a0d9 WatchSource:0}: Error finding container 2070a42c5b15ec1f36e93d061ddccc97a8388a50ab7b3277c82fc52179f4a0d9: Status 404 returned error can't find the container with id 2070a42c5b15ec1f36e93d061ddccc97a8388a50ab7b3277c82fc52179f4a0d9 Oct 30 21:44:17.020533 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:17.020405 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9"] Oct 30 21:44:17.437219 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:17.437178 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9" event={"ID":"9cfebe73-9386-47d7-9d93-46e0f38b5b71","Type":"ContainerStarted","Data":"123f6b73e2bfb54f3da5ee58a1d8a38fbe6e996fe902e78093571f612c3bb1eb"} Oct 30 21:44:17.437750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:17.437242 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9" event={"ID":"9cfebe73-9386-47d7-9d93-46e0f38b5b71","Type":"ContainerStarted","Data":"2070a42c5b15ec1f36e93d061ddccc97a8388a50ab7b3277c82fc52179f4a0d9"} Oct 30 21:44:17.487391 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:17.487334 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9" podStartSLOduration=1.4873188480000001 podStartE2EDuration="1.487318848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:17.48533051 +0000 UTC m=+2644.606404270" watchObservedRunningTime="2026-10-30 21:44:17.487318848 +0000 UTC m=+2644.608392593" Oct 30 21:44:18.052013 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:18.051972 4926 scope.go:122] "RemoveContainer" containerID="c423a11dafa84665c14793b8c51dcc67649f33a31da6d61c55390439f504d06a" Oct 30 21:44:18.064787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:18.064765 4926 scope.go:122] "RemoveContainer" containerID="31d29ef57e62151ad655b0cc348085d1421ecf8b2de7adb70a584d14a7721b2d" Oct 30 21:44:18.076564 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:18.076534 4926 scope.go:122] "RemoveContainer" containerID="a6a95324e5b584b5e60cc03c3623e91f2f612d744d11d32778ac91e33dda563c" Oct 30 21:44:18.088842 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:18.088817 4926 scope.go:122] "RemoveContainer" containerID="ac77fe218d5e9f4e9031b63926cdd34e6ca8ad945c480799882346de419f6990" Oct 30 21:44:18.098950 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:18.098929 4926 scope.go:122] "RemoveContainer" containerID="507e4109b3ff2893141ad527d690b3ff8e98672eb614a74be6ce481c5f1b9269" Oct 30 21:44:18.446433 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:18.446396 4926 generic.go:356] "Generic (PLEG): container finished" podID="9cfebe73-9386-47d7-9d93-46e0f38b5b71" containerID="123f6b73e2bfb54f3da5ee58a1d8a38fbe6e996fe902e78093571f612c3bb1eb" exitCode=0 Oct 30 21:44:18.446924 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:18.446485 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9" event={"ID":"9cfebe73-9386-47d7-9d93-46e0f38b5b71","Type":"ContainerDied","Data":"123f6b73e2bfb54f3da5ee58a1d8a38fbe6e996fe902e78093571f612c3bb1eb"} Oct 30 21:44:19.651058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:19.651014 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9" Oct 30 21:44:19.760396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:19.760298 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cfebe73-9386-47d7-9d93-46e0f38b5b71-kube-api-access-5vfpv\" (UniqueName: \"kubernetes.io/projected/9cfebe73-9386-47d7-9d93-46e0f38b5b71-kube-api-access-5vfpv\") pod \"9cfebe73-9386-47d7-9d93-46e0f38b5b71\" (UID: \"9cfebe73-9386-47d7-9d93-46e0f38b5b71\") " Oct 30 21:44:19.760585 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:19.760450 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cfebe73-9386-47d7-9d93-46e0f38b5b71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cfebe73-9386-47d7-9d93-46e0f38b5b71-maas-api-xks-ca\") pod \"9cfebe73-9386-47d7-9d93-46e0f38b5b71\" (UID: \"9cfebe73-9386-47d7-9d93-46e0f38b5b71\") " Oct 30 21:44:19.760651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:19.760638 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cfebe73-9386-47d7-9d93-46e0f38b5b71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cfebe73-9386-47d7-9d93-46e0f38b5b71-maas-api-openshift-ca\") pod \"9cfebe73-9386-47d7-9d93-46e0f38b5b71\" (UID: \"9cfebe73-9386-47d7-9d93-46e0f38b5b71\") " Oct 30 21:44:19.761098 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:19.761065 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cfebe73-9386-47d7-9d93-46e0f38b5b71-maas-api-openshift-ca" pod "9cfebe73-9386-47d7-9d93-46e0f38b5b71" (UID: "9cfebe73-9386-47d7-9d93-46e0f38b5b71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:19.762677 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:19.762641 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cfebe73-9386-47d7-9d93-46e0f38b5b71-maas-api-xks-ca" pod "9cfebe73-9386-47d7-9d93-46e0f38b5b71" (UID: "9cfebe73-9386-47d7-9d93-46e0f38b5b71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:19.762677 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:19.762656 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cfebe73-9386-47d7-9d93-46e0f38b5b71-kube-api-access-5vfpv" pod "9cfebe73-9386-47d7-9d93-46e0f38b5b71" (UID: "9cfebe73-9386-47d7-9d93-46e0f38b5b71"). InnerVolumeSpecName "kube-api-access-5vfpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:19.862877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:19.862833 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cfebe73-9386-47d7-9d93-46e0f38b5b71-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:44:19.862877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:19.862872 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cfebe73-9386-47d7-9d93-46e0f38b5b71-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:44:19.862877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:19.862883 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vfpv\" (UniqueName: \"kubernetes.io/projected/9cfebe73-9386-47d7-9d93-46e0f38b5b71-kube-api-access-5vfpv\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:44:20.464683 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:20.464645 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9" Oct 30 21:44:20.464683 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:20.464656 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9" event={"ID":"9cfebe73-9386-47d7-9d93-46e0f38b5b71","Type":"ContainerDied","Data":"2070a42c5b15ec1f36e93d061ddccc97a8388a50ab7b3277c82fc52179f4a0d9"} Oct 30 21:44:20.465248 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:20.464722 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2070a42c5b15ec1f36e93d061ddccc97a8388a50ab7b3277c82fc52179f4a0d9" Oct 30 21:44:26.727441 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:26.727355 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23a1a48b-222bl"] Oct 30 21:44:26.727926 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:26.727912 4926 cpu_manager.go:414] "containerMap: removing container" podUID="9cfebe73-9386-47d7-9d93-46e0f38b5b71" containerName="revoke-keys" Oct 30 21:44:26.727972 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:26.727927 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cfebe73-9386-47d7-9d93-46e0f38b5b71" containerName="revoke-keys" Oct 30 21:44:26.728075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:26.728065 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cfebe73-9386-47d7-9d93-46e0f38b5b71" containerName="revoke-keys" Oct 30 21:44:26.732079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:26.732043 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23a1a48b-222bl" Oct 30 21:44:26.746691 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:26.746662 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:44:26.757907 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:26.757871 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:44:26.762825 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:26.762793 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23a1a48b-222bl"] Oct 30 21:44:26.863130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:26.863083 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54d68ed2-dfbd-4aea-98ec-d182e78c2267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23a1a48b-222bl\" (UID: \"54d68ed2-dfbd-4aea-98ec-d182e78c2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23a1a48b-222bl" Oct 30 21:44:26.863363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:26.863282 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54d68ed2-dfbd-4aea-98ec-d182e78c2267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23a1a48b-222bl\" (UID: \"54d68ed2-dfbd-4aea-98ec-d182e78c2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23a1a48b-222bl" Oct 30 21:44:26.863443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:26.863417 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqmq9\" (UniqueName: \"kubernetes.io/projected/54d68ed2-dfbd-4aea-98ec-d182e78c2267-kube-api-access-sqmq9\") pod \"maas-api-revoke-sub-23a1a48b-222bl\" (UID: \"54d68ed2-dfbd-4aea-98ec-d182e78c2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23a1a48b-222bl" Oct 30 21:44:26.964777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:26.964738 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54d68ed2-dfbd-4aea-98ec-d182e78c2267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23a1a48b-222bl\" (UID: \"54d68ed2-dfbd-4aea-98ec-d182e78c2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23a1a48b-222bl" Oct 30 21:44:26.965000 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:26.964817 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqmq9\" (UniqueName: \"kubernetes.io/projected/54d68ed2-dfbd-4aea-98ec-d182e78c2267-kube-api-access-sqmq9\") pod \"maas-api-revoke-sub-23a1a48b-222bl\" (UID: \"54d68ed2-dfbd-4aea-98ec-d182e78c2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23a1a48b-222bl" Oct 30 21:44:26.965000 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:26.964943 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54d68ed2-dfbd-4aea-98ec-d182e78c2267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23a1a48b-222bl\" (UID: \"54d68ed2-dfbd-4aea-98ec-d182e78c2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23a1a48b-222bl" Oct 30 21:44:26.965552 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:26.965525 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54d68ed2-dfbd-4aea-98ec-d182e78c2267-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23a1a48b-222bl\" (UID: \"54d68ed2-dfbd-4aea-98ec-d182e78c2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23a1a48b-222bl" Oct 30 21:44:26.967409 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:26.967381 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54d68ed2-dfbd-4aea-98ec-d182e78c2267-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23a1a48b-222bl\" (UID: \"54d68ed2-dfbd-4aea-98ec-d182e78c2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23a1a48b-222bl" Oct 30 21:44:26.988192 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:26.988115 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqmq9\" (UniqueName: \"kubernetes.io/projected/54d68ed2-dfbd-4aea-98ec-d182e78c2267-kube-api-access-sqmq9\") pod \"maas-api-revoke-sub-23a1a48b-222bl\" (UID: \"54d68ed2-dfbd-4aea-98ec-d182e78c2267\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23a1a48b-222bl" Oct 30 21:44:27.047849 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:27.047805 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23a1a48b-222bl" Oct 30 21:44:27.363646 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:27.363610 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23a1a48b-222bl"] Oct 30 21:44:27.364174 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:44:27.364148 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d68ed2_dfbd_4aea_98ec_d182e78c2267.slice/crio-6cbf7b2d0a0d8ba8a60026418574b6852e219ff84727ff02170899c4aeeea947 WatchSource:0}: Error finding container 6cbf7b2d0a0d8ba8a60026418574b6852e219ff84727ff02170899c4aeeea947: Status 404 returned error can't find the container with id 6cbf7b2d0a0d8ba8a60026418574b6852e219ff84727ff02170899c4aeeea947 Oct 30 21:44:27.534109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:27.534062 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23a1a48b-222bl" event={"ID":"54d68ed2-dfbd-4aea-98ec-d182e78c2267","Type":"ContainerStarted","Data":"6cbf7b2d0a0d8ba8a60026418574b6852e219ff84727ff02170899c4aeeea947"} Oct 30 21:44:28.542756 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:28.542714 4926 generic.go:356] "Generic (PLEG): container finished" podID="54d68ed2-dfbd-4aea-98ec-d182e78c2267" containerID="d55b08ce2a5ee76f879c413f3076b1a5f1a5171785f2036c3dc6f6c17a628219" exitCode=0 Oct 30 21:44:28.543238 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:28.542804 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23a1a48b-222bl" event={"ID":"54d68ed2-dfbd-4aea-98ec-d182e78c2267","Type":"ContainerDied","Data":"d55b08ce2a5ee76f879c413f3076b1a5f1a5171785f2036c3dc6f6c17a628219"} Oct 30 21:44:29.746115 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:29.746087 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23a1a48b-222bl" Oct 30 21:44:29.913983 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:29.913953 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54d68ed2-dfbd-4aea-98ec-d182e78c2267-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54d68ed2-dfbd-4aea-98ec-d182e78c2267-maas-api-xks-ca\") pod \"54d68ed2-dfbd-4aea-98ec-d182e78c2267\" (UID: \"54d68ed2-dfbd-4aea-98ec-d182e78c2267\") " Oct 30 21:44:29.914215 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:29.914065 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54d68ed2-dfbd-4aea-98ec-d182e78c2267-kube-api-access-sqmq9\" (UniqueName: \"kubernetes.io/projected/54d68ed2-dfbd-4aea-98ec-d182e78c2267-kube-api-access-sqmq9\") pod \"54d68ed2-dfbd-4aea-98ec-d182e78c2267\" (UID: \"54d68ed2-dfbd-4aea-98ec-d182e78c2267\") " Oct 30 21:44:29.914215 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:29.914187 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54d68ed2-dfbd-4aea-98ec-d182e78c2267-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54d68ed2-dfbd-4aea-98ec-d182e78c2267-maas-api-openshift-ca\") pod \"54d68ed2-dfbd-4aea-98ec-d182e78c2267\" (UID: \"54d68ed2-dfbd-4aea-98ec-d182e78c2267\") " Oct 30 21:44:29.914640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:29.914612 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54d68ed2-dfbd-4aea-98ec-d182e78c2267-maas-api-openshift-ca" pod "54d68ed2-dfbd-4aea-98ec-d182e78c2267" (UID: "54d68ed2-dfbd-4aea-98ec-d182e78c2267"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:29.914822 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:29.914795 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54d68ed2-dfbd-4aea-98ec-d182e78c2267-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:44:29.916403 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:29.916380 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54d68ed2-dfbd-4aea-98ec-d182e78c2267-maas-api-xks-ca" pod "54d68ed2-dfbd-4aea-98ec-d182e78c2267" (UID: "54d68ed2-dfbd-4aea-98ec-d182e78c2267"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:29.916505 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:29.916429 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d68ed2-dfbd-4aea-98ec-d182e78c2267-kube-api-access-sqmq9" pod "54d68ed2-dfbd-4aea-98ec-d182e78c2267" (UID: "54d68ed2-dfbd-4aea-98ec-d182e78c2267"). InnerVolumeSpecName "kube-api-access-sqmq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:30.016254 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:30.016218 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54d68ed2-dfbd-4aea-98ec-d182e78c2267-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:44:30.016254 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:30.016250 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqmq9\" (UniqueName: \"kubernetes.io/projected/54d68ed2-dfbd-4aea-98ec-d182e78c2267-kube-api-access-sqmq9\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:44:30.560884 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:30.560851 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23a1a48b-222bl" Oct 30 21:44:30.561129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:30.560896 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23a1a48b-222bl" event={"ID":"54d68ed2-dfbd-4aea-98ec-d182e78c2267","Type":"ContainerDied","Data":"6cbf7b2d0a0d8ba8a60026418574b6852e219ff84727ff02170899c4aeeea947"} Oct 30 21:44:30.561129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:30.560949 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cbf7b2d0a0d8ba8a60026418574b6852e219ff84727ff02170899c4aeeea947" Oct 30 21:44:34.550956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:34.550917 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz"] Oct 30 21:44:34.551456 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:34.551181 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" podUID="c60ed799-e6b9-464d-87d3-9f46bdfc7fcc" containerName="discovery" containerID="cri-o://c738ef7e1f84497e196df302f943f63912a80170c5238e14c045dc070953f6c2" gracePeriod=30 Oct 30 21:44:34.865853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:34.865820 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:44:34.913054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:34.912491 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-csr-ca-configmap\") pod \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " Oct 30 21:44:34.913054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:34.912949 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-csr-ca-configmap" pod "c60ed799-e6b9-464d-87d3-9f46bdfc7fcc" (UID: "c60ed799-e6b9-464d-87d3-9f46bdfc7fcc"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:34.913319 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:34.913154 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-cacerts\" (UniqueName: \"kubernetes.io/secret/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-cacerts\") pod \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " Oct 30 21:44:34.918075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:34.913710 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-local-certs\") pod \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " Oct 30 21:44:34.918075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:34.914126 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-csr-dns-cert\") pod \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " Oct 30 21:44:34.918075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:34.914471 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-token\" (UniqueName: \"kubernetes.io/projected/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-token\") pod \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " Oct 30 21:44:34.918075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:34.914849 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-kube-api-access-d775j\" (UniqueName: \"kubernetes.io/projected/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-kube-api-access-d775j\") pod \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " Oct 30 21:44:34.918075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:34.915219 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-kubeconfig\") pod \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\" (UID: \"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc\") " Oct 30 21:44:34.918075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:34.915895 4926 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-csr-ca-configmap\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:44:34.918616 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:34.918531 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-kubeconfig" pod "c60ed799-e6b9-464d-87d3-9f46bdfc7fcc" (UID: "c60ed799-e6b9-464d-87d3-9f46bdfc7fcc"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:34.924055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:34.924002 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-local-certs" pod "c60ed799-e6b9-464d-87d3-9f46bdfc7fcc" (UID: "c60ed799-e6b9-464d-87d3-9f46bdfc7fcc"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:34.924224 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:34.924017 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-csr-dns-cert" pod "c60ed799-e6b9-464d-87d3-9f46bdfc7fcc" (UID: "c60ed799-e6b9-464d-87d3-9f46bdfc7fcc"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:34.924751 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:34.924723 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-token" pod "c60ed799-e6b9-464d-87d3-9f46bdfc7fcc" (UID: "c60ed799-e6b9-464d-87d3-9f46bdfc7fcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:34.925826 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:34.925777 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-kube-api-access-d775j" pod "c60ed799-e6b9-464d-87d3-9f46bdfc7fcc" (UID: "c60ed799-e6b9-464d-87d3-9f46bdfc7fcc"). InnerVolumeSpecName "kube-api-access-d775j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:34.930221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:34.930195 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-cacerts" pod "c60ed799-e6b9-464d-87d3-9f46bdfc7fcc" (UID: "c60ed799-e6b9-464d-87d3-9f46bdfc7fcc"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:35.017508 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:35.017472 4926 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-local-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:44:35.017508 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:35.017504 4926 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-csr-dns-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:44:35.017508 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:35.017514 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:44:35.017771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:35.017523 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d775j\" (UniqueName: \"kubernetes.io/projected/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-kube-api-access-d775j\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:44:35.017771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:35.017532 4926 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-istio-kubeconfig\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:44:35.017771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:35.017541 4926 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc-cacerts\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:44:35.627986 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:35.627950 4926 generic.go:356] "Generic (PLEG): container finished" podID="c60ed799-e6b9-464d-87d3-9f46bdfc7fcc" containerID="c738ef7e1f84497e196df302f943f63912a80170c5238e14c045dc070953f6c2" exitCode=0 Oct 30 21:44:35.628446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:35.628014 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" Oct 30 21:44:35.628446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:35.628064 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" event={"ID":"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc","Type":"ContainerDied","Data":"c738ef7e1f84497e196df302f943f63912a80170c5238e14c045dc070953f6c2"} Oct 30 21:44:35.628446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:35.628119 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" event={"ID":"c60ed799-e6b9-464d-87d3-9f46bdfc7fcc","Type":"ContainerDied","Data":"0c61e970451a6344e31bf98be2d57f3e004e98e43cae539b0665b75e801cf853"} Oct 30 21:44:35.628446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:35.628140 4926 scope.go:122] "RemoveContainer" containerID="c738ef7e1f84497e196df302f943f63912a80170c5238e14c045dc070953f6c2" Oct 30 21:44:35.644141 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:35.644120 4926 scope.go:122] "RemoveContainer" containerID="c738ef7e1f84497e196df302f943f63912a80170c5238e14c045dc070953f6c2" Oct 30 21:44:35.644639 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:44:35.644606 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c738ef7e1f84497e196df302f943f63912a80170c5238e14c045dc070953f6c2\": container with ID starting with c738ef7e1f84497e196df302f943f63912a80170c5238e14c045dc070953f6c2 not found: ID does not exist" containerID="c738ef7e1f84497e196df302f943f63912a80170c5238e14c045dc070953f6c2" Oct 30 21:44:35.644773 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:35.644651 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c738ef7e1f84497e196df302f943f63912a80170c5238e14c045dc070953f6c2"} err="failed to get container status \"c738ef7e1f84497e196df302f943f63912a80170c5238e14c045dc070953f6c2\": rpc error: code = NotFound desc = could not find container \"c738ef7e1f84497e196df302f943f63912a80170c5238e14c045dc070953f6c2\": container with ID starting with c738ef7e1f84497e196df302f943f63912a80170c5238e14c045dc070953f6c2 not found: ID does not exist" Oct 30 21:44:35.701140 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:35.701098 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz"] Oct 30 21:44:35.708416 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:35.708378 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz"] Oct 30 21:44:37.018176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:37.018140 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c60ed799-e6b9-464d-87d3-9f46bdfc7fcc" path="/var/lib/kubelet/pods/c60ed799-e6b9-464d-87d3-9f46bdfc7fcc/volumes" Oct 30 21:44:37.487567 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:44:37.487528 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:44:39.241169 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:39.236362 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x"] Oct 30 21:44:39.763501 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:39.763450 4926 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8s4mz container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.114:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:44:39.763745 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:39.763544 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s4mz" podUID="c60ed799-e6b9-464d-87d3-9f46bdfc7fcc" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.114:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:44:46.322573 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:46.322528 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:44:51.395794 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:51.395746 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7404-de416a"] Oct 30 21:44:51.396639 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:51.396616 4926 cpu_manager.go:414] "containerMap: removing container" podUID="54d68ed2-dfbd-4aea-98ec-d182e78c2267" containerName="revoke-keys" Oct 30 21:44:51.396726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:51.396642 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54d68ed2-dfbd-4aea-98ec-d182e78c2267" containerName="revoke-keys" Oct 30 21:44:51.396726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:51.396707 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c60ed799-e6b9-464d-87d3-9f46bdfc7fcc" containerName="discovery" Oct 30 21:44:51.396726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:51.396715 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c60ed799-e6b9-464d-87d3-9f46bdfc7fcc" containerName="discovery" Oct 30 21:44:51.396853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:51.396841 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="54d68ed2-dfbd-4aea-98ec-d182e78c2267" containerName="revoke-keys" Oct 30 21:44:51.396890 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:51.396867 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c60ed799-e6b9-464d-87d3-9f46bdfc7fcc" containerName="discovery" Oct 30 21:44:51.400965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:51.400942 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7404-de416a" Oct 30 21:44:51.414220 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:51.414194 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7404-de416a"] Oct 30 21:44:51.418205 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:51.418177 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gxp2m\"" Oct 30 21:44:51.564960 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:51.564920 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kptmg\" (UniqueName: \"kubernetes.io/projected/1d9809fc-fca7-4ab1-b216-46718ba69a11-kube-api-access-kptmg\") pod \"test-curl-7404-de416a\" (UID: \"1d9809fc-fca7-4ab1-b216-46718ba69a11\") " pod="opendatahub/test-curl-7404-de416a" Oct 30 21:44:51.666973 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:51.666921 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kptmg\" (UniqueName: \"kubernetes.io/projected/1d9809fc-fca7-4ab1-b216-46718ba69a11-kube-api-access-kptmg\") pod \"test-curl-7404-de416a\" (UID: \"1d9809fc-fca7-4ab1-b216-46718ba69a11\") " pod="opendatahub/test-curl-7404-de416a" Oct 30 21:44:51.685889 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:51.685857 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kptmg\" (UniqueName: \"kubernetes.io/projected/1d9809fc-fca7-4ab1-b216-46718ba69a11-kube-api-access-kptmg\") pod \"test-curl-7404-de416a\" (UID: \"1d9809fc-fca7-4ab1-b216-46718ba69a11\") " pod="opendatahub/test-curl-7404-de416a" Oct 30 21:44:51.715246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:51.715210 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7404-de416a" Oct 30 21:44:52.045831 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:44:52.045798 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d9809fc_fca7_4ab1_b216_46718ba69a11.slice/crio-736b7229cdf263697e17e0d37f5a72347f5facebee6c62a6ff5236e12241e50c WatchSource:0}: Error finding container 736b7229cdf263697e17e0d37f5a72347f5facebee6c62a6ff5236e12241e50c: Status 404 returned error can't find the container with id 736b7229cdf263697e17e0d37f5a72347f5facebee6c62a6ff5236e12241e50c Oct 30 21:44:52.046279 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.046248 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7404-de416a"] Oct 30 21:44:52.047933 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.047917 4926 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:44:52.786873 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.786823 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7404-de416a" event={"ID":"1d9809fc-fca7-4ab1-b216-46718ba69a11","Type":"ContainerStarted","Data":"736b7229cdf263697e17e0d37f5a72347f5facebee6c62a6ff5236e12241e50c"} Oct 30 21:44:52.874010 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.873968 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs"] Oct 30 21:44:52.879429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.879403 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:52.887464 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.887432 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-workload-certs\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:52.887639 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.887583 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49c7689-1388-4442-8ea8-825da60fab98-istio-token\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:52.887712 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.887667 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-istio-envoy\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:52.887712 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.887692 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49c7689-1388-4442-8ea8-825da60fab98-istiod-ca-cert\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:52.887826 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.887793 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-credential-socket\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:52.888004 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.887910 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kktpw\" (UniqueName: \"kubernetes.io/projected/c49c7689-1388-4442-8ea8-825da60fab98-kube-api-access-kktpw\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:52.888185 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.888109 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49c7689-1388-4442-8ea8-825da60fab98-istio-podinfo\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:52.888255 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.888184 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-workload-socket\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:52.888255 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.888207 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-istio-data\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:52.908840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.908805 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-c3bbba-openshift-default-dockercfg-jnfwq\"" Oct 30 21:44:52.926388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.926352 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs"] Oct 30 21:44:53.002056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.989676 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-workload-socket\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:53.002056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.989730 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-istio-data\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:53.002056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.989804 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-workload-certs\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:53.002056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.989881 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49c7689-1388-4442-8ea8-825da60fab98-istio-token\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:53.002056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.989951 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-istio-envoy\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:53.002056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.999460 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-workload-socket\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:53.002056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:52.999729 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-workload-certs\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:53.014955 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:53.009128 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49c7689-1388-4442-8ea8-825da60fab98-istiod-ca-cert\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:53.014955 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:53.009212 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-credential-socket\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:53.014955 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:53.009334 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kktpw\" (UniqueName: \"kubernetes.io/projected/c49c7689-1388-4442-8ea8-825da60fab98-kube-api-access-kktpw\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:53.014955 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:53.009378 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49c7689-1388-4442-8ea8-825da60fab98-istio-podinfo\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:53.014955 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:53.012954 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49c7689-1388-4442-8ea8-825da60fab98-istiod-ca-cert\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:53.014955 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:53.013262 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-credential-socket\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:53.027545 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:53.019892 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-istio-data\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:53.150101 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:53.149925 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-istio-envoy\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:53.161096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:53.150490 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49c7689-1388-4442-8ea8-825da60fab98-istio-podinfo\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:53.161096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:53.160830 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49c7689-1388-4442-8ea8-825da60fab98-istio-token\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:53.174301 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:53.174229 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kktpw\" (UniqueName: \"kubernetes.io/projected/c49c7689-1388-4442-8ea8-825da60fab98-kube-api-access-kktpw\") pod \"e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:53.205083 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:53.200088 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:53.617292 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:44:53.617256 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49c7689_1388_4442_8ea8_825da60fab98.slice/crio-7f8407245e0cb68e7188389a1ce4aa3d6907a8a0aef61d3505761c3defee42dc WatchSource:0}: Error finding container 7f8407245e0cb68e7188389a1ce4aa3d6907a8a0aef61d3505761c3defee42dc: Status 404 returned error can't find the container with id 7f8407245e0cb68e7188389a1ce4aa3d6907a8a0aef61d3505761c3defee42dc Oct 30 21:44:53.619987 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:53.619960 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs"] Oct 30 21:44:53.621210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:53.621166 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:44:53.621326 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:53.621246 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:44:53.621326 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:53.621287 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:44:53.823781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:53.821378 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" event={"ID":"c49c7689-1388-4442-8ea8-825da60fab98","Type":"ContainerStarted","Data":"2b102897365236e348c1ba81bd40f11f89da75a791b377b6132bb10e0554394d"} Oct 30 21:44:53.823781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:53.821476 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" event={"ID":"c49c7689-1388-4442-8ea8-825da60fab98","Type":"ContainerStarted","Data":"7f8407245e0cb68e7188389a1ce4aa3d6907a8a0aef61d3505761c3defee42dc"} Oct 30 21:44:53.875103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:53.874182 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" podStartSLOduration=1.874160511 podStartE2EDuration="1.874160511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:53.873397805 +0000 UTC m=+2680.994471552" watchObservedRunningTime="2026-10-30 21:44:53.874160511 +0000 UTC m=+2680.995234257" Oct 30 21:44:54.202162 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.201278 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:54.203453 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.203270 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 21:44:54.203453 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.203329 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" podUID="c49c7689-1388-4442-8ea8-825da60fab98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 21:44:54.265075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.263694 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" podUID="32325e5c-9e22-4068-a425-a668cf4b9f04" containerName="main" containerID="cri-o://7741a61c7a6de7710bf2b59d10eb0f9246f188c4c2fc16413bd52cb2e1e419f7" gracePeriod=45 Oct 30 21:44:54.622399 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.622369 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:44:54.641999 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.641965 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-tmp-dir\") pod \"32325e5c-9e22-4068-a425-a668cf4b9f04\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " Oct 30 21:44:54.642208 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.642102 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-model-cache\") pod \"32325e5c-9e22-4068-a425-a668cf4b9f04\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " Oct 30 21:44:54.642208 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.642167 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-home\" (UniqueName: \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-home\") pod \"32325e5c-9e22-4068-a425-a668cf4b9f04\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " Oct 30 21:44:54.642307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.642217 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-dshm\" (UniqueName: \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-dshm\") pod \"32325e5c-9e22-4068-a425-a668cf4b9f04\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " Oct 30 21:44:54.642307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.642251 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-tmp-dir" pod "32325e5c-9e22-4068-a425-a668cf4b9f04" (UID: "32325e5c-9e22-4068-a425-a668cf4b9f04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:54.642307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.642266 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32325e5c-9e22-4068-a425-a668cf4b9f04-tls-certs\" (UniqueName: \"kubernetes.io/secret/32325e5c-9e22-4068-a425-a668cf4b9f04-tls-certs\") pod \"32325e5c-9e22-4068-a425-a668cf4b9f04\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " Oct 30 21:44:54.642435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.642417 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32325e5c-9e22-4068-a425-a668cf4b9f04-kube-api-access-5kvpq\" (UniqueName: \"kubernetes.io/projected/32325e5c-9e22-4068-a425-a668cf4b9f04-kube-api-access-5kvpq\") pod \"32325e5c-9e22-4068-a425-a668cf4b9f04\" (UID: \"32325e5c-9e22-4068-a425-a668cf4b9f04\") " Oct 30 21:44:54.642796 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.642770 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-home" pod "32325e5c-9e22-4068-a425-a668cf4b9f04" (UID: "32325e5c-9e22-4068-a425-a668cf4b9f04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:54.643103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.643079 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:44:54.643181 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.643105 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:44:54.643181 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.643118 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-model-cache" pod "32325e5c-9e22-4068-a425-a668cf4b9f04" (UID: "32325e5c-9e22-4068-a425-a668cf4b9f04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:54.651733 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.651664 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-dshm" pod "32325e5c-9e22-4068-a425-a668cf4b9f04" (UID: "32325e5c-9e22-4068-a425-a668cf4b9f04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:54.651932 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.651827 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32325e5c-9e22-4068-a425-a668cf4b9f04-tls-certs" pod "32325e5c-9e22-4068-a425-a668cf4b9f04" (UID: "32325e5c-9e22-4068-a425-a668cf4b9f04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:54.653793 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.653765 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32325e5c-9e22-4068-a425-a668cf4b9f04-kube-api-access-5kvpq" pod "32325e5c-9e22-4068-a425-a668cf4b9f04" (UID: "32325e5c-9e22-4068-a425-a668cf4b9f04"). InnerVolumeSpecName "kube-api-access-5kvpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:54.744649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.744570 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:44:54.744649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.744613 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32325e5c-9e22-4068-a425-a668cf4b9f04-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:44:54.744649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.744627 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32325e5c-9e22-4068-a425-a668cf4b9f04-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:44:54.744649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.744640 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kvpq\" (UniqueName: \"kubernetes.io/projected/32325e5c-9e22-4068-a425-a668cf4b9f04-kube-api-access-5kvpq\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:44:54.833829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.833790 4926 generic.go:356] "Generic (PLEG): container finished" podID="32325e5c-9e22-4068-a425-a668cf4b9f04" containerID="7741a61c7a6de7710bf2b59d10eb0f9246f188c4c2fc16413bd52cb2e1e419f7" exitCode=0 Oct 30 21:44:54.834342 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.834144 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" Oct 30 21:44:54.834525 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.834501 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" event={"ID":"32325e5c-9e22-4068-a425-a668cf4b9f04","Type":"ContainerDied","Data":"7741a61c7a6de7710bf2b59d10eb0f9246f188c4c2fc16413bd52cb2e1e419f7"} Oct 30 21:44:54.834591 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.834546 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x" event={"ID":"32325e5c-9e22-4068-a425-a668cf4b9f04","Type":"ContainerDied","Data":"4c3249b7b5d02cdc1f09ce1f140d6e33a54b91d9f5a4b66b2ae1054cb6623163"} Oct 30 21:44:54.834591 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.834568 4926 scope.go:122] "RemoveContainer" containerID="7741a61c7a6de7710bf2b59d10eb0f9246f188c4c2fc16413bd52cb2e1e419f7" Oct 30 21:44:54.863576 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.863552 4926 scope.go:122] "RemoveContainer" containerID="7741a61c7a6de7710bf2b59d10eb0f9246f188c4c2fc16413bd52cb2e1e419f7" Oct 30 21:44:54.864004 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:44:54.863973 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7741a61c7a6de7710bf2b59d10eb0f9246f188c4c2fc16413bd52cb2e1e419f7\": container with ID starting with 7741a61c7a6de7710bf2b59d10eb0f9246f188c4c2fc16413bd52cb2e1e419f7 not found: ID does not exist" containerID="7741a61c7a6de7710bf2b59d10eb0f9246f188c4c2fc16413bd52cb2e1e419f7" Oct 30 21:44:54.864108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.864017 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7741a61c7a6de7710bf2b59d10eb0f9246f188c4c2fc16413bd52cb2e1e419f7"} err="failed to get container status \"7741a61c7a6de7710bf2b59d10eb0f9246f188c4c2fc16413bd52cb2e1e419f7\": rpc error: code = NotFound desc = could not find container \"7741a61c7a6de7710bf2b59d10eb0f9246f188c4c2fc16413bd52cb2e1e419f7\": container with ID starting with 7741a61c7a6de7710bf2b59d10eb0f9246f188c4c2fc16413bd52cb2e1e419f7 not found: ID does not exist" Oct 30 21:44:54.896000 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.895945 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x"] Oct 30 21:44:54.902309 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:54.902275 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-908469/auth-test-model-908469-kserve-7d97d589d5-kxh2x"] Oct 30 21:44:55.024492 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:55.024455 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32325e5c-9e22-4068-a425-a668cf4b9f04" path="/var/lib/kubelet/pods/32325e5c-9e22-4068-a425-a668cf4b9f04/volumes" Oct 30 21:44:55.207785 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:55.207750 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:55.844267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:55.844214 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7404-de416a" event={"ID":"1d9809fc-fca7-4ab1-b216-46718ba69a11","Type":"ContainerStarted","Data":"dc8c3f48d468b000aaf20e5391c273dd41e4a4efd70c15273ace40bace93246d"} Oct 30 21:44:55.844696 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:55.844627 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:55.845465 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:55.845443 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:44:55.895558 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:55.895485 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7404-de416a" podStartSLOduration=1.721724652 podStartE2EDuration="4.895466611s" totalImagesPullingTime="3.173741959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:55.888895089 +0000 UTC m=+2683.009968837" watchObservedRunningTime="2026-10-30 21:44:55.895466611 +0000 UTC m=+2683.016540359" Oct 30 21:44:56.828651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:56.828609 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7404-de416a"] Oct 30 21:44:56.835058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:56.835011 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7404-de416a"] Oct 30 21:44:57.079170 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.079084 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-641b550a-rlr56"] Oct 30 21:44:57.079730 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.079713 4926 cpu_manager.go:414] "containerMap: removing container" podUID="1d9809fc-fca7-4ab1-b216-46718ba69a11" containerName="test-curl-7404-de416a" Oct 30 21:44:57.079773 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.079732 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9809fc-fca7-4ab1-b216-46718ba69a11" containerName="test-curl-7404-de416a" Oct 30 21:44:57.079773 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.079765 4926 cpu_manager.go:414] "containerMap: removing container" podUID="32325e5c-9e22-4068-a425-a668cf4b9f04" containerName="main" Oct 30 21:44:57.079773 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.079770 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32325e5c-9e22-4068-a425-a668cf4b9f04" containerName="main" Oct 30 21:44:57.079878 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.079872 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9809fc-fca7-4ab1-b216-46718ba69a11" containerName="test-curl-7404-de416a" Oct 30 21:44:57.079912 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.079885 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="32325e5c-9e22-4068-a425-a668cf4b9f04" containerName="main" Oct 30 21:44:57.083615 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.083596 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641b550a-rlr56" Oct 30 21:44:57.115153 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.115111 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-641b550a-rlr56"] Oct 30 21:44:57.118327 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.118297 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:44:57.118502 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.118388 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:44:57.199849 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.199806 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-641b550a-rlr56\" (UID: \"66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641b550a-rlr56" Oct 30 21:44:57.200056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.199985 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-641b550a-rlr56\" (UID: \"66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641b550a-rlr56" Oct 30 21:44:57.200180 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.200161 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss96x\" (UniqueName: \"kubernetes.io/projected/66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f-kube-api-access-ss96x\") pod \"maas-api-revoke-sub-641b550a-rlr56\" (UID: \"66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641b550a-rlr56" Oct 30 21:44:57.301512 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.301461 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-641b550a-rlr56\" (UID: \"66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641b550a-rlr56" Oct 30 21:44:57.301707 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.301624 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss96x\" (UniqueName: \"kubernetes.io/projected/66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f-kube-api-access-ss96x\") pod \"maas-api-revoke-sub-641b550a-rlr56\" (UID: \"66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641b550a-rlr56" Oct 30 21:44:57.301765 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.301749 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-641b550a-rlr56\" (UID: \"66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641b550a-rlr56" Oct 30 21:44:57.302383 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.302356 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-641b550a-rlr56\" (UID: \"66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641b550a-rlr56" Oct 30 21:44:57.304214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.304193 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-641b550a-rlr56\" (UID: \"66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641b550a-rlr56" Oct 30 21:44:57.326376 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.326346 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss96x\" (UniqueName: \"kubernetes.io/projected/66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f-kube-api-access-ss96x\") pod \"maas-api-revoke-sub-641b550a-rlr56\" (UID: \"66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-641b550a-rlr56" Oct 30 21:44:57.403150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.403020 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641b550a-rlr56" Oct 30 21:44:57.723068 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:44:57.723020 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a5f0b5_aff3_43d6_b25a_acbcc3ab1f0f.slice/crio-f12b73216aa33c8e7ab8eab06476f8887a54b6c0e04605b8422d7b51c133017d WatchSource:0}: Error finding container f12b73216aa33c8e7ab8eab06476f8887a54b6c0e04605b8422d7b51c133017d: Status 404 returned error can't find the container with id f12b73216aa33c8e7ab8eab06476f8887a54b6c0e04605b8422d7b51c133017d Oct 30 21:44:57.724736 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.724707 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-641b550a-rlr56"] Oct 30 21:44:57.862176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.862141 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641b550a-rlr56" event={"ID":"66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f","Type":"ContainerStarted","Data":"f12b73216aa33c8e7ab8eab06476f8887a54b6c0e04605b8422d7b51c133017d"} Oct 30 21:44:57.862365 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:57.862343 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7404-de416a" podUID="1d9809fc-fca7-4ab1-b216-46718ba69a11" containerName="test-curl-7404-de416a" containerID="cri-o://dc8c3f48d468b000aaf20e5391c273dd41e4a4efd70c15273ace40bace93246d" gracePeriod=2 Oct 30 21:44:58.218660 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.218620 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7"] Oct 30 21:44:58.223056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.223020 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7" Oct 30 21:44:58.247522 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.247490 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7"] Oct 30 21:44:58.254409 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.253978 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c3bbba\"" Oct 30 21:44:58.254409 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.254052 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c3bbba-dockercfg-8s9dm\"" Oct 30 21:44:58.331062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.325835 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fmgk\" (UniqueName: \"kubernetes.io/projected/dd40b019-c8a9-4ceb-ae90-efa4eba7331f-kube-api-access-9fmgk\") pod \"payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7\" (UID: \"dd40b019-c8a9-4ceb-ae90-efa4eba7331f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7" Oct 30 21:44:58.331062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.325927 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd40b019-c8a9-4ceb-ae90-efa4eba7331f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7\" (UID: \"dd40b019-c8a9-4ceb-ae90-efa4eba7331f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7" Oct 30 21:44:58.411123 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.387235 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt"] Oct 30 21:44:58.420274 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.420233 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt"] Oct 30 21:44:58.420445 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.420373 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt" Oct 30 21:44:58.429060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.428770 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fmgk\" (UniqueName: \"kubernetes.io/projected/dd40b019-c8a9-4ceb-ae90-efa4eba7331f-kube-api-access-9fmgk\") pod \"payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7\" (UID: \"dd40b019-c8a9-4ceb-ae90-efa4eba7331f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7" Oct 30 21:44:58.429060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.428862 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd40b019-c8a9-4ceb-ae90-efa4eba7331f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7\" (UID: \"dd40b019-c8a9-4ceb-ae90-efa4eba7331f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7" Oct 30 21:44:58.435103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.434930 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd40b019-c8a9-4ceb-ae90-efa4eba7331f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7\" (UID: \"dd40b019-c8a9-4ceb-ae90-efa4eba7331f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7" Oct 30 21:44:58.484093 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.468635 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fmgk\" (UniqueName: \"kubernetes.io/projected/dd40b019-c8a9-4ceb-ae90-efa4eba7331f-kube-api-access-9fmgk\") pod \"payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7\" (UID: \"dd40b019-c8a9-4ceb-ae90-efa4eba7331f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7" Oct 30 21:44:58.535369 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.535332 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptqbw\" (UniqueName: \"kubernetes.io/projected/d5aa7f00-cf51-434d-bb02-a14f937f9f8e-kube-api-access-ptqbw\") pod \"payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt\" (UID: \"d5aa7f00-cf51-434d-bb02-a14f937f9f8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt" Oct 30 21:44:58.535562 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.535383 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5aa7f00-cf51-434d-bb02-a14f937f9f8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt\" (UID: \"d5aa7f00-cf51-434d-bb02-a14f937f9f8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt" Oct 30 21:44:58.537350 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.537325 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7" Oct 30 21:44:58.637405 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.637365 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptqbw\" (UniqueName: \"kubernetes.io/projected/d5aa7f00-cf51-434d-bb02-a14f937f9f8e-kube-api-access-ptqbw\") pod \"payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt\" (UID: \"d5aa7f00-cf51-434d-bb02-a14f937f9f8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt" Oct 30 21:44:58.637613 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.637437 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5aa7f00-cf51-434d-bb02-a14f937f9f8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt\" (UID: \"d5aa7f00-cf51-434d-bb02-a14f937f9f8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt" Oct 30 21:44:58.638716 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.638691 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5aa7f00-cf51-434d-bb02-a14f937f9f8e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt\" (UID: \"d5aa7f00-cf51-434d-bb02-a14f937f9f8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt" Oct 30 21:44:58.658179 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.658143 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptqbw\" (UniqueName: \"kubernetes.io/projected/d5aa7f00-cf51-434d-bb02-a14f937f9f8e-kube-api-access-ptqbw\") pod \"payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt\" (UID: \"d5aa7f00-cf51-434d-bb02-a14f937f9f8e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt" Oct 30 21:44:58.750011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.749912 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt" Oct 30 21:44:58.858347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.858312 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7"] Oct 30 21:44:58.859922 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:44:58.859865 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd40b019_c8a9_4ceb_ae90_efa4eba7331f.slice/crio-2fbd2f5abce63f07600efffbbc6edff935a1fbded95e0eed25c2392c42cb474b WatchSource:0}: Error finding container 2fbd2f5abce63f07600efffbbc6edff935a1fbded95e0eed25c2392c42cb474b: Status 404 returned error can't find the container with id 2fbd2f5abce63f07600efffbbc6edff935a1fbded95e0eed25c2392c42cb474b Oct 30 21:44:58.883447 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.883408 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7" event={"ID":"dd40b019-c8a9-4ceb-ae90-efa4eba7331f","Type":"ContainerStarted","Data":"2fbd2f5abce63f07600efffbbc6edff935a1fbded95e0eed25c2392c42cb474b"} Oct 30 21:44:58.885854 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.885783 4926 generic.go:356] "Generic (PLEG): container finished" podID="66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f" containerID="50743f30bb624e802d80c628d7ee14261cf97d693943ec95f9d2d1baeea115b3" exitCode=0 Oct 30 21:44:58.885993 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:58.885968 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641b550a-rlr56" event={"ID":"66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f","Type":"ContainerDied","Data":"50743f30bb624e802d80c628d7ee14261cf97d693943ec95f9d2d1baeea115b3"} Oct 30 21:44:59.068009 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:44:59.067947 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5aa7f00_cf51_434d_bb02_a14f937f9f8e.slice/crio-b5fd662dc280faa6fa6c36e999eaed94f2c4defa2f26d1066615081e7287e9e7 WatchSource:0}: Error finding container b5fd662dc280faa6fa6c36e999eaed94f2c4defa2f26d1066615081e7287e9e7: Status 404 returned error can't find the container with id b5fd662dc280faa6fa6c36e999eaed94f2c4defa2f26d1066615081e7287e9e7 Oct 30 21:44:59.069789 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:59.069754 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt"] Oct 30 21:44:59.897826 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:59.897786 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt" event={"ID":"d5aa7f00-cf51-434d-bb02-a14f937f9f8e","Type":"ContainerStarted","Data":"1b55d02ddf3f0b65626fe6f63d23bdee2857803c00869e81e349629a52325dea"} Oct 30 21:44:59.898259 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:59.897857 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt" event={"ID":"d5aa7f00-cf51-434d-bb02-a14f937f9f8e","Type":"ContainerStarted","Data":"b5fd662dc280faa6fa6c36e999eaed94f2c4defa2f26d1066615081e7287e9e7"} Oct 30 21:44:59.898259 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:59.897884 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt" Oct 30 21:44:59.900428 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:59.900394 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7" event={"ID":"dd40b019-c8a9-4ceb-ae90-efa4eba7331f","Type":"ContainerStarted","Data":"c17d4d99192af125f2aa4d1ed366e8ba008e1e0430fc41b14a0a119965905b82"} Oct 30 21:44:59.954458 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:59.954391 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt" podStartSLOduration=1.9543698649999999 podStartE2EDuration="1.954369865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:59.953284698 +0000 UTC m=+2687.074358456" watchObservedRunningTime="2026-10-30 21:44:59.954369865 +0000 UTC m=+2687.075443612" Oct 30 21:44:59.997902 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:44:59.997191 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7" podStartSLOduration=1.997167944 podStartE2EDuration="1.997167944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:59.993931056 +0000 UTC m=+2687.115004803" watchObservedRunningTime="2026-10-30 21:44:59.997167944 +0000 UTC m=+2687.118241692" Oct 30 21:45:00.130081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.129938 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641b550a-rlr56" Oct 30 21:45:00.135955 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.135921 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7404-de416a" Oct 30 21:45:00.185161 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.185116 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2"] Oct 30 21:45:00.185991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.185967 4926 cpu_manager.go:414] "containerMap: removing container" podUID="66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f" containerName="revoke-keys" Oct 30 21:45:00.186116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.185993 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f" containerName="revoke-keys" Oct 30 21:45:00.186253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.186232 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f" containerName="revoke-keys" Oct 30 21:45:00.190257 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.190235 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2" Oct 30 21:45:00.194281 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.194251 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95"] Oct 30 21:45:00.208263 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.208235 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95" Oct 30 21:45:00.215152 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.214882 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj"] Oct 30 21:45:00.221330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.220273 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj" Oct 30 21:45:00.226242 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.226164 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm"] Oct 30 21:45:00.242956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.242929 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds"] Oct 30 21:45:00.243138 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.243059 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm" Oct 30 21:45:00.248673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.248643 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95"] Oct 30 21:45:00.250254 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.248691 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2"] Oct 30 21:45:00.250254 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.248748 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds" Oct 30 21:45:00.256718 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.256636 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj"] Oct 30 21:45:00.262990 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.261640 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d9809fc-fca7-4ab1-b216-46718ba69a11-kube-api-access-kptmg\" (UniqueName: \"kubernetes.io/projected/1d9809fc-fca7-4ab1-b216-46718ba69a11-kube-api-access-kptmg\") pod \"1d9809fc-fca7-4ab1-b216-46718ba69a11\" (UID: \"1d9809fc-fca7-4ab1-b216-46718ba69a11\") " Oct 30 21:45:00.262990 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.261691 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f-kube-api-access-ss96x\" (UniqueName: \"kubernetes.io/projected/66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f-kube-api-access-ss96x\") pod \"66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f\" (UID: \"66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f\") " Oct 30 21:45:00.262990 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.261727 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f-maas-api-xks-ca\") pod \"66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f\" (UID: \"66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f\") " Oct 30 21:45:00.262990 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.261914 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f-maas-api-openshift-ca\") pod \"66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f\" (UID: \"66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f\") " Oct 30 21:45:00.262990 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.262245 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwqwp\" (UniqueName: \"kubernetes.io/projected/547cf37c-665f-4c4b-ab6f-06b5422b9cc9-kube-api-access-hwqwp\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2\" (UID: \"547cf37c-665f-4c4b-ab6f-06b5422b9cc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2" Oct 30 21:45:00.262990 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.262288 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/547cf37c-665f-4c4b-ab6f-06b5422b9cc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2\" (UID: \"547cf37c-665f-4c4b-ab6f-06b5422b9cc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2" Oct 30 21:45:00.264305 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.264247 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f-maas-api-openshift-ca" pod "66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f" (UID: "66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:00.264808 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.264779 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f-kube-api-access-ss96x" pod "66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f" (UID: "66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f"). InnerVolumeSpecName "kube-api-access-ss96x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:00.265401 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.265370 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f-maas-api-xks-ca" pod "66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f" (UID: "66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:00.265900 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.265873 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds"] Oct 30 21:45:00.272235 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.272194 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9809fc-fca7-4ab1-b216-46718ba69a11-kube-api-access-kptmg" pod "1d9809fc-fca7-4ab1-b216-46718ba69a11" (UID: "1d9809fc-fca7-4ab1-b216-46718ba69a11"). InnerVolumeSpecName "kube-api-access-kptmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:00.317799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.317751 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm"] Oct 30 21:45:00.324298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.324257 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t"] Oct 30 21:45:00.328659 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.328633 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t" Oct 30 21:45:00.332114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.332083 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k"] Oct 30 21:45:00.337657 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.337369 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k" Oct 30 21:45:00.339116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.339089 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn"] Oct 30 21:45:00.346151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.346117 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4"] Oct 30 21:45:00.346504 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.346379 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn" Oct 30 21:45:00.351710 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.351687 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k"] Oct 30 21:45:00.351853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.351793 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4" Oct 30 21:45:00.352781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.352747 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t"] Oct 30 21:45:00.364169 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.364130 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/596e496a-35f5-483f-bbde-62e3dc028766-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds\" (UID: \"596e496a-35f5-483f-bbde-62e3dc028766\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds" Oct 30 21:45:00.364315 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.364215 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwqf6\" (UniqueName: \"kubernetes.io/projected/bd055827-04e0-4946-bf79-b8e59e81fa21-kube-api-access-mwqf6\") pod \"maas-api-key-cleanup-29889945-5sb95\" (UID: \"bd055827-04e0-4946-bf79-b8e59e81fa21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95" Oct 30 21:45:00.364315 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.364253 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/205a1101-2a99-4800-84d8-56a39d3690b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm\" (UID: \"205a1101-2a99-4800-84d8-56a39d3690b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm" Oct 30 21:45:00.364315 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.364286 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/657d2744-b23d-4dbb-a75d-23fea5f9a286-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj\" (UID: \"657d2744-b23d-4dbb-a75d-23fea5f9a286\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj" Oct 30 21:45:00.364463 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.364385 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm4gl\" (UniqueName: \"kubernetes.io/projected/657d2744-b23d-4dbb-a75d-23fea5f9a286-kube-api-access-rm4gl\") pod \"maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj\" (UID: \"657d2744-b23d-4dbb-a75d-23fea5f9a286\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj" Oct 30 21:45:00.364463 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.364445 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbmzd\" (UniqueName: \"kubernetes.io/projected/596e496a-35f5-483f-bbde-62e3dc028766-kube-api-access-rbmzd\") pod \"maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds\" (UID: \"596e496a-35f5-483f-bbde-62e3dc028766\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds" Oct 30 21:45:00.364608 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.364584 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd055827-04e0-4946-bf79-b8e59e81fa21-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-5sb95\" (UID: \"bd055827-04e0-4946-bf79-b8e59e81fa21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95" Oct 30 21:45:00.364669 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.364653 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwqwp\" (UniqueName: \"kubernetes.io/projected/547cf37c-665f-4c4b-ab6f-06b5422b9cc9-kube-api-access-hwqwp\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2\" (UID: \"547cf37c-665f-4c4b-ab6f-06b5422b9cc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2" Oct 30 21:45:00.364735 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.364688 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/547cf37c-665f-4c4b-ab6f-06b5422b9cc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2\" (UID: \"547cf37c-665f-4c4b-ab6f-06b5422b9cc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2" Oct 30 21:45:00.364798 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.364747 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnbxr\" (UniqueName: \"kubernetes.io/projected/205a1101-2a99-4800-84d8-56a39d3690b6-kube-api-access-jnbxr\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm\" (UID: \"205a1101-2a99-4800-84d8-56a39d3690b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm" Oct 30 21:45:00.364866 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.364850 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kptmg\" (UniqueName: \"kubernetes.io/projected/1d9809fc-fca7-4ab1-b216-46718ba69a11-kube-api-access-kptmg\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:00.364937 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.364874 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss96x\" (UniqueName: \"kubernetes.io/projected/66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f-kube-api-access-ss96x\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:00.364937 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.364890 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:00.364937 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.364906 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:00.366108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.365948 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/547cf37c-665f-4c4b-ab6f-06b5422b9cc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2\" (UID: \"547cf37c-665f-4c4b-ab6f-06b5422b9cc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2" Oct 30 21:45:00.366108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.365998 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz"] Oct 30 21:45:00.374903 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.374832 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t"] Oct 30 21:45:00.377365 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.377222 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz" Oct 30 21:45:00.379953 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.379933 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4"] Oct 30 21:45:00.380126 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.380070 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t" Oct 30 21:45:00.398435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.398409 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4"] Oct 30 21:45:00.403979 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.403950 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwqwp\" (UniqueName: \"kubernetes.io/projected/547cf37c-665f-4c4b-ab6f-06b5422b9cc9-kube-api-access-hwqwp\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2\" (UID: \"547cf37c-665f-4c4b-ab6f-06b5422b9cc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2" Oct 30 21:45:00.404218 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.404015 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4" Oct 30 21:45:00.428277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.426231 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn"] Oct 30 21:45:00.432245 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.432089 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t"] Oct 30 21:45:00.438553 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.438155 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz"] Oct 30 21:45:00.449116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.448138 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n"] Oct 30 21:45:00.455116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.455087 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n" Oct 30 21:45:00.461325 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.461298 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4"] Oct 30 21:45:00.466674 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.466622 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd055827-04e0-4946-bf79-b8e59e81fa21-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-5sb95\" (UID: \"bd055827-04e0-4946-bf79-b8e59e81fa21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95" Oct 30 21:45:00.466830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.466744 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30def38d-8a67-4281-9031-c6f7f7c96bc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4\" (UID: \"30def38d-8a67-4281-9031-c6f7f7c96bc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4" Oct 30 21:45:00.466830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.466785 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz\" (UID: \"42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz" Oct 30 21:45:00.471346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.466838 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5g6m\" (UniqueName: \"kubernetes.io/projected/2a03d85d-cd11-4d9a-ae67-29ced159f159-kube-api-access-z5g6m\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k\" (UID: \"2a03d85d-cd11-4d9a-ae67-29ced159f159\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k" Oct 30 21:45:00.471346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.466874 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f27f31-e62c-482e-bfba-10c2250410a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4\" (UID: \"d0f27f31-e62c-482e-bfba-10c2250410a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4" Oct 30 21:45:00.471346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.466926 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnbxr\" (UniqueName: \"kubernetes.io/projected/205a1101-2a99-4800-84d8-56a39d3690b6-kube-api-access-jnbxr\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm\" (UID: \"205a1101-2a99-4800-84d8-56a39d3690b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm" Oct 30 21:45:00.471346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.466988 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00eb47b-3c81-4420-a948-45dae9800548-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t\" (UID: \"f00eb47b-3c81-4420-a948-45dae9800548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t" Oct 30 21:45:00.471346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.467028 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brfrf\" (UniqueName: \"kubernetes.io/projected/c61bf7ef-856d-4d2e-b4ca-402028fabd8f-kube-api-access-brfrf\") pod \"maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t\" (UID: \"c61bf7ef-856d-4d2e-b4ca-402028fabd8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t" Oct 30 21:45:00.471346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.467144 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a03d85d-cd11-4d9a-ae67-29ced159f159-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k\" (UID: \"2a03d85d-cd11-4d9a-ae67-29ced159f159\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k" Oct 30 21:45:00.471346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.467178 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19dafab6-6daa-4e31-bc89-3ebd04adffbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn\" (UID: \"19dafab6-6daa-4e31-bc89-3ebd04adffbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn" Oct 30 21:45:00.471346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.467246 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/596e496a-35f5-483f-bbde-62e3dc028766-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds\" (UID: \"596e496a-35f5-483f-bbde-62e3dc028766\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds" Oct 30 21:45:00.471346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.467295 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9rbf\" (UniqueName: \"kubernetes.io/projected/f00eb47b-3c81-4420-a948-45dae9800548-kube-api-access-w9rbf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t\" (UID: \"f00eb47b-3c81-4420-a948-45dae9800548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t" Oct 30 21:45:00.471346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.467371 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwqf6\" (UniqueName: \"kubernetes.io/projected/bd055827-04e0-4946-bf79-b8e59e81fa21-kube-api-access-mwqf6\") pod \"maas-api-key-cleanup-29889945-5sb95\" (UID: \"bd055827-04e0-4946-bf79-b8e59e81fa21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95" Oct 30 21:45:00.471346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.467416 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/205a1101-2a99-4800-84d8-56a39d3690b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm\" (UID: \"205a1101-2a99-4800-84d8-56a39d3690b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm" Oct 30 21:45:00.471346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.467459 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnlk9\" (UniqueName: \"kubernetes.io/projected/30def38d-8a67-4281-9031-c6f7f7c96bc5-kube-api-access-qnlk9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4\" (UID: \"30def38d-8a67-4281-9031-c6f7f7c96bc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4" Oct 30 21:45:00.471346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.467502 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/657d2744-b23d-4dbb-a75d-23fea5f9a286-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj\" (UID: \"657d2744-b23d-4dbb-a75d-23fea5f9a286\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj" Oct 30 21:45:00.471346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.467545 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61bf7ef-856d-4d2e-b4ca-402028fabd8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t\" (UID: \"c61bf7ef-856d-4d2e-b4ca-402028fabd8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t" Oct 30 21:45:00.471346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.467614 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm4gl\" (UniqueName: \"kubernetes.io/projected/657d2744-b23d-4dbb-a75d-23fea5f9a286-kube-api-access-rm4gl\") pod \"maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj\" (UID: \"657d2744-b23d-4dbb-a75d-23fea5f9a286\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj" Oct 30 21:45:00.471346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.467700 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7g67\" (UniqueName: \"kubernetes.io/projected/42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543-kube-api-access-b7g67\") pod \"maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz\" (UID: \"42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz" Oct 30 21:45:00.471346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.467738 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwnsh\" (UniqueName: \"kubernetes.io/projected/19dafab6-6daa-4e31-bc89-3ebd04adffbb-kube-api-access-rwnsh\") pod \"maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn\" (UID: \"19dafab6-6daa-4e31-bc89-3ebd04adffbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn" Oct 30 21:45:00.471346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.467803 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbmzd\" (UniqueName: \"kubernetes.io/projected/596e496a-35f5-483f-bbde-62e3dc028766-kube-api-access-rbmzd\") pod \"maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds\" (UID: \"596e496a-35f5-483f-bbde-62e3dc028766\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds" Oct 30 21:45:00.471346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.467891 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgmwx\" (UniqueName: \"kubernetes.io/projected/d0f27f31-e62c-482e-bfba-10c2250410a4-kube-api-access-fgmwx\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4\" (UID: \"d0f27f31-e62c-482e-bfba-10c2250410a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4" Oct 30 21:45:00.471346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.469005 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd055827-04e0-4946-bf79-b8e59e81fa21-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-5sb95\" (UID: \"bd055827-04e0-4946-bf79-b8e59e81fa21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95" Oct 30 21:45:00.471346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.470242 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/596e496a-35f5-483f-bbde-62e3dc028766-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds\" (UID: \"596e496a-35f5-483f-bbde-62e3dc028766\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds" Oct 30 21:45:00.471346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.471116 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/205a1101-2a99-4800-84d8-56a39d3690b6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm\" (UID: \"205a1101-2a99-4800-84d8-56a39d3690b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm" Oct 30 21:45:00.472652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.471953 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/657d2744-b23d-4dbb-a75d-23fea5f9a286-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj\" (UID: \"657d2744-b23d-4dbb-a75d-23fea5f9a286\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj" Oct 30 21:45:00.504415 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.504374 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz"] Oct 30 21:45:00.508519 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.508266 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2" Oct 30 21:45:00.524248 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.524222 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n"] Oct 30 21:45:00.524455 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.524340 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz" Oct 30 21:45:00.532713 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.532674 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm4gl\" (UniqueName: \"kubernetes.io/projected/657d2744-b23d-4dbb-a75d-23fea5f9a286-kube-api-access-rm4gl\") pod \"maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj\" (UID: \"657d2744-b23d-4dbb-a75d-23fea5f9a286\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj" Oct 30 21:45:00.534066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.533712 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbmzd\" (UniqueName: \"kubernetes.io/projected/596e496a-35f5-483f-bbde-62e3dc028766-kube-api-access-rbmzd\") pod \"maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds\" (UID: \"596e496a-35f5-483f-bbde-62e3dc028766\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds" Oct 30 21:45:00.535014 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.534963 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwqf6\" (UniqueName: \"kubernetes.io/projected/bd055827-04e0-4946-bf79-b8e59e81fa21-kube-api-access-mwqf6\") pod \"maas-api-key-cleanup-29889945-5sb95\" (UID: \"bd055827-04e0-4946-bf79-b8e59e81fa21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95" Oct 30 21:45:00.536665 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.536490 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95" Oct 30 21:45:00.541626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.541591 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p"] Oct 30 21:45:00.544497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.544324 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj" Oct 30 21:45:00.546112 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.546088 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnbxr\" (UniqueName: \"kubernetes.io/projected/205a1101-2a99-4800-84d8-56a39d3690b6-kube-api-access-jnbxr\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm\" (UID: \"205a1101-2a99-4800-84d8-56a39d3690b6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm" Oct 30 21:45:00.547820 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.547792 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p" Oct 30 21:45:00.565350 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.565317 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz"] Oct 30 21:45:00.565505 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.565330 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm" Oct 30 21:45:00.576462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.570656 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30def38d-8a67-4281-9031-c6f7f7c96bc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4\" (UID: \"30def38d-8a67-4281-9031-c6f7f7c96bc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4" Oct 30 21:45:00.576462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.570706 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz\" (UID: \"42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz" Oct 30 21:45:00.576462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.570743 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5g6m\" (UniqueName: \"kubernetes.io/projected/2a03d85d-cd11-4d9a-ae67-29ced159f159-kube-api-access-z5g6m\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k\" (UID: \"2a03d85d-cd11-4d9a-ae67-29ced159f159\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k" Oct 30 21:45:00.576462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.570780 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f27f31-e62c-482e-bfba-10c2250410a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4\" (UID: \"d0f27f31-e62c-482e-bfba-10c2250410a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4" Oct 30 21:45:00.576462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.570832 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00eb47b-3c81-4420-a948-45dae9800548-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t\" (UID: \"f00eb47b-3c81-4420-a948-45dae9800548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t" Oct 30 21:45:00.576462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.570869 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brfrf\" (UniqueName: \"kubernetes.io/projected/c61bf7ef-856d-4d2e-b4ca-402028fabd8f-kube-api-access-brfrf\") pod \"maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t\" (UID: \"c61bf7ef-856d-4d2e-b4ca-402028fabd8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t" Oct 30 21:45:00.576462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.570933 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a03d85d-cd11-4d9a-ae67-29ced159f159-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k\" (UID: \"2a03d85d-cd11-4d9a-ae67-29ced159f159\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k" Oct 30 21:45:00.576462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.570963 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19dafab6-6daa-4e31-bc89-3ebd04adffbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn\" (UID: \"19dafab6-6daa-4e31-bc89-3ebd04adffbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn" Oct 30 21:45:00.576462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.570991 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bb7714-22f3-4fe2-887c-71d638b4d61d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n\" (UID: \"b0bb7714-22f3-4fe2-887c-71d638b4d61d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n" Oct 30 21:45:00.576462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.571065 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rbf\" (UniqueName: \"kubernetes.io/projected/f00eb47b-3c81-4420-a948-45dae9800548-kube-api-access-w9rbf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t\" (UID: \"f00eb47b-3c81-4420-a948-45dae9800548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t" Oct 30 21:45:00.576462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.571100 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfm2c\" (UniqueName: \"kubernetes.io/projected/b0bb7714-22f3-4fe2-887c-71d638b4d61d-kube-api-access-cfm2c\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n\" (UID: \"b0bb7714-22f3-4fe2-887c-71d638b4d61d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n" Oct 30 21:45:00.576462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.571163 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnlk9\" (UniqueName: \"kubernetes.io/projected/30def38d-8a67-4281-9031-c6f7f7c96bc5-kube-api-access-qnlk9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4\" (UID: \"30def38d-8a67-4281-9031-c6f7f7c96bc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4" Oct 30 21:45:00.576462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.571199 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61bf7ef-856d-4d2e-b4ca-402028fabd8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t\" (UID: \"c61bf7ef-856d-4d2e-b4ca-402028fabd8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t" Oct 30 21:45:00.576462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.571282 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7g67\" (UniqueName: \"kubernetes.io/projected/42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543-kube-api-access-b7g67\") pod \"maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz\" (UID: \"42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz" Oct 30 21:45:00.576462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.571316 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwnsh\" (UniqueName: \"kubernetes.io/projected/19dafab6-6daa-4e31-bc89-3ebd04adffbb-kube-api-access-rwnsh\") pod \"maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn\" (UID: \"19dafab6-6daa-4e31-bc89-3ebd04adffbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn" Oct 30 21:45:00.576462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.571395 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgmwx\" (UniqueName: \"kubernetes.io/projected/d0f27f31-e62c-482e-bfba-10c2250410a4-kube-api-access-fgmwx\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4\" (UID: \"d0f27f31-e62c-482e-bfba-10c2250410a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4" Oct 30 21:45:00.576462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.572339 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30def38d-8a67-4281-9031-c6f7f7c96bc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4\" (UID: \"30def38d-8a67-4281-9031-c6f7f7c96bc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4" Oct 30 21:45:00.576462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.572766 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19dafab6-6daa-4e31-bc89-3ebd04adffbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn\" (UID: \"19dafab6-6daa-4e31-bc89-3ebd04adffbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn" Oct 30 21:45:00.576462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.573854 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a03d85d-cd11-4d9a-ae67-29ced159f159-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k\" (UID: \"2a03d85d-cd11-4d9a-ae67-29ced159f159\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k" Oct 30 21:45:00.576462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.574567 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz\" (UID: \"42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz" Oct 30 21:45:00.576462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.575827 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f27f31-e62c-482e-bfba-10c2250410a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4\" (UID: \"d0f27f31-e62c-482e-bfba-10c2250410a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4" Oct 30 21:45:00.578996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.576646 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61bf7ef-856d-4d2e-b4ca-402028fabd8f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t\" (UID: \"c61bf7ef-856d-4d2e-b4ca-402028fabd8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t" Oct 30 21:45:00.578996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.578101 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00eb47b-3c81-4420-a948-45dae9800548-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t\" (UID: \"f00eb47b-3c81-4420-a948-45dae9800548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t" Oct 30 21:45:00.602783 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.602751 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p"] Oct 30 21:45:00.603352 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.603318 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds" Oct 30 21:45:00.630129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.626061 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7g67\" (UniqueName: \"kubernetes.io/projected/42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543-kube-api-access-b7g67\") pod \"maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz\" (UID: \"42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz" Oct 30 21:45:00.630129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.626296 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brfrf\" (UniqueName: \"kubernetes.io/projected/c61bf7ef-856d-4d2e-b4ca-402028fabd8f-kube-api-access-brfrf\") pod \"maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t\" (UID: \"c61bf7ef-856d-4d2e-b4ca-402028fabd8f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t" Oct 30 21:45:00.630129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.626722 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgmwx\" (UniqueName: \"kubernetes.io/projected/d0f27f31-e62c-482e-bfba-10c2250410a4-kube-api-access-fgmwx\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4\" (UID: \"d0f27f31-e62c-482e-bfba-10c2250410a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4" Oct 30 21:45:00.630129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.627308 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnlk9\" (UniqueName: \"kubernetes.io/projected/30def38d-8a67-4281-9031-c6f7f7c96bc5-kube-api-access-qnlk9\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4\" (UID: \"30def38d-8a67-4281-9031-c6f7f7c96bc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4" Oct 30 21:45:00.630129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.627810 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwnsh\" (UniqueName: \"kubernetes.io/projected/19dafab6-6daa-4e31-bc89-3ebd04adffbb-kube-api-access-rwnsh\") pod \"maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn\" (UID: \"19dafab6-6daa-4e31-bc89-3ebd04adffbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn" Oct 30 21:45:00.631365 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.631303 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9rbf\" (UniqueName: \"kubernetes.io/projected/f00eb47b-3c81-4420-a948-45dae9800548-kube-api-access-w9rbf\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t\" (UID: \"f00eb47b-3c81-4420-a948-45dae9800548\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t" Oct 30 21:45:00.636898 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.636811 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5g6m\" (UniqueName: \"kubernetes.io/projected/2a03d85d-cd11-4d9a-ae67-29ced159f159-kube-api-access-z5g6m\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k\" (UID: \"2a03d85d-cd11-4d9a-ae67-29ced159f159\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k" Oct 30 21:45:00.663232 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.651910 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t" Oct 30 21:45:00.681574 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.681522 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bb7714-22f3-4fe2-887c-71d638b4d61d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n\" (UID: \"b0bb7714-22f3-4fe2-887c-71d638b4d61d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n" Oct 30 21:45:00.681769 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.681599 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfm2c\" (UniqueName: \"kubernetes.io/projected/b0bb7714-22f3-4fe2-887c-71d638b4d61d-kube-api-access-cfm2c\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n\" (UID: \"b0bb7714-22f3-4fe2-887c-71d638b4d61d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n" Oct 30 21:45:00.681769 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.681649 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c51dea80-dea4-4195-8c0a-8f28ea265669-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p\" (UID: \"c51dea80-dea4-4195-8c0a-8f28ea265669\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p" Oct 30 21:45:00.681895 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.681800 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz\" (UID: \"e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz" Oct 30 21:45:00.681963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.681931 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75tlf\" (UniqueName: \"kubernetes.io/projected/e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3-kube-api-access-75tlf\") pod \"maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz\" (UID: \"e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz" Oct 30 21:45:00.682023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.681986 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lpqv\" (UniqueName: \"kubernetes.io/projected/c51dea80-dea4-4195-8c0a-8f28ea265669-kube-api-access-7lpqv\") pod \"maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p\" (UID: \"c51dea80-dea4-4195-8c0a-8f28ea265669\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p" Oct 30 21:45:00.682341 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.682312 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k" Oct 30 21:45:00.682863 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.682834 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bb7714-22f3-4fe2-887c-71d638b4d61d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n\" (UID: \"b0bb7714-22f3-4fe2-887c-71d638b4d61d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n" Oct 30 21:45:00.718080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.702394 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4" Oct 30 21:45:00.718080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.705284 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn" Oct 30 21:45:00.718781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.718640 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfm2c\" (UniqueName: \"kubernetes.io/projected/b0bb7714-22f3-4fe2-887c-71d638b4d61d-kube-api-access-cfm2c\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n\" (UID: \"b0bb7714-22f3-4fe2-887c-71d638b4d61d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n" Oct 30 21:45:00.759000 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.758888 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz" Oct 30 21:45:00.783314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.783097 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t" Oct 30 21:45:00.795853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.794449 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75tlf\" (UniqueName: \"kubernetes.io/projected/e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3-kube-api-access-75tlf\") pod \"maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz\" (UID: \"e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz" Oct 30 21:45:00.795853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.794517 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lpqv\" (UniqueName: \"kubernetes.io/projected/c51dea80-dea4-4195-8c0a-8f28ea265669-kube-api-access-7lpqv\") pod \"maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p\" (UID: \"c51dea80-dea4-4195-8c0a-8f28ea265669\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p" Oct 30 21:45:00.795853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.794609 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c51dea80-dea4-4195-8c0a-8f28ea265669-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p\" (UID: \"c51dea80-dea4-4195-8c0a-8f28ea265669\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p" Oct 30 21:45:00.795853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.794733 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz\" (UID: \"e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz" Oct 30 21:45:00.795853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.795526 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz\" (UID: \"e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz" Oct 30 21:45:00.797483 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.796483 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c51dea80-dea4-4195-8c0a-8f28ea265669-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p\" (UID: \"c51dea80-dea4-4195-8c0a-8f28ea265669\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p" Oct 30 21:45:00.810914 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.810683 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4" Oct 30 21:45:00.813559 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.813483 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n" Oct 30 21:45:00.865111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.865068 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lpqv\" (UniqueName: \"kubernetes.io/projected/c51dea80-dea4-4195-8c0a-8f28ea265669-kube-api-access-7lpqv\") pod \"maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p\" (UID: \"c51dea80-dea4-4195-8c0a-8f28ea265669\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p" Oct 30 21:45:00.920178 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.920108 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75tlf\" (UniqueName: \"kubernetes.io/projected/e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3-kube-api-access-75tlf\") pod \"maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz\" (UID: \"e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz" Oct 30 21:45:00.959774 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.955472 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2"] Oct 30 21:45:00.969047 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.969006 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p" Oct 30 21:45:00.992767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.991791 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641b550a-rlr56" event={"ID":"66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f","Type":"ContainerDied","Data":"f12b73216aa33c8e7ab8eab06476f8887a54b6c0e04605b8422d7b51c133017d"} Oct 30 21:45:00.992767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.991858 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f12b73216aa33c8e7ab8eab06476f8887a54b6c0e04605b8422d7b51c133017d" Oct 30 21:45:00.992767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:00.991961 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-641b550a-rlr56" Oct 30 21:45:01.003210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:01.002912 4926 generic.go:356] "Generic (PLEG): container finished" podID="1d9809fc-fca7-4ab1-b216-46718ba69a11" containerID="dc8c3f48d468b000aaf20e5391c273dd41e4a4efd70c15273ace40bace93246d" exitCode=137 Oct 30 21:45:01.004451 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:01.003406 4926 scope.go:122] "RemoveContainer" containerID="dc8c3f48d468b000aaf20e5391c273dd41e4a4efd70c15273ace40bace93246d" Oct 30 21:45:01.004451 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:01.003599 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7404-de416a" Oct 30 21:45:01.004451 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:01.004396 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7" Oct 30 21:45:01.035183 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:01.035153 4926 scope.go:122] "RemoveContainer" containerID="dc8c3f48d468b000aaf20e5391c273dd41e4a4efd70c15273ace40bace93246d" Oct 30 21:45:01.046800 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:01.046756 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc8c3f48d468b000aaf20e5391c273dd41e4a4efd70c15273ace40bace93246d\": container with ID starting with dc8c3f48d468b000aaf20e5391c273dd41e4a4efd70c15273ace40bace93246d not found: ID does not exist" containerID="dc8c3f48d468b000aaf20e5391c273dd41e4a4efd70c15273ace40bace93246d" Oct 30 21:45:01.047021 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:01.046947 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc8c3f48d468b000aaf20e5391c273dd41e4a4efd70c15273ace40bace93246d"} err="failed to get container status \"dc8c3f48d468b000aaf20e5391c273dd41e4a4efd70c15273ace40bace93246d\": rpc error: code = NotFound desc = could not find container \"dc8c3f48d468b000aaf20e5391c273dd41e4a4efd70c15273ace40bace93246d\": container with ID starting with dc8c3f48d468b000aaf20e5391c273dd41e4a4efd70c15273ace40bace93246d not found: ID does not exist" Oct 30 21:45:01.071391 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:01.066847 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d9809fc-fca7-4ab1-b216-46718ba69a11" path="/var/lib/kubelet/pods/1d9809fc-fca7-4ab1-b216-46718ba69a11/volumes" Oct 30 21:45:01.208682 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:01.208561 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz" Oct 30 21:45:01.384771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:01.384746 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95"] Oct 30 21:45:01.390819 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:01.390355 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm"] Oct 30 21:45:01.396298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:01.396223 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj"] Oct 30 21:45:01.579859 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:01.579825 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k"] Oct 30 21:45:01.588239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:01.588203 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n"] Oct 30 21:45:01.594281 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:01.594209 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4"] Oct 30 21:45:01.600951 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:01.600926 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t"] Oct 30 21:45:01.605177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:01.605153 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t"] Oct 30 21:45:01.611315 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:01.611286 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz"] Oct 30 21:45:01.618430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:01.618360 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds"] Oct 30 21:45:01.623684 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:01.623660 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn"] Oct 30 21:45:01.629367 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:01.629341 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4"] Oct 30 21:45:01.637267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:01.637246 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p"] Oct 30 21:45:01.675558 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:45:01.675514 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a03d85d_cd11_4d9a_ae67_29ced159f159.slice/crio-34dd91abe0deb24548bf5f937062ba40f8cb35a98e9ff17754c0e67aa4fae8c0 WatchSource:0}: Error finding container 34dd91abe0deb24548bf5f937062ba40f8cb35a98e9ff17754c0e67aa4fae8c0: Status 404 returned error can't find the container with id 34dd91abe0deb24548bf5f937062ba40f8cb35a98e9ff17754c0e67aa4fae8c0 Oct 30 21:45:01.677888 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:45:01.677847 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30def38d_8a67_4281_9031_c6f7f7c96bc5.slice/crio-d54d5710d71fde2ba37668627a22cf8cb780578b64660b0f7ef8e8af69ef2a79 WatchSource:0}: Error finding container d54d5710d71fde2ba37668627a22cf8cb780578b64660b0f7ef8e8af69ef2a79: Status 404 returned error can't find the container with id d54d5710d71fde2ba37668627a22cf8cb780578b64660b0f7ef8e8af69ef2a79 Oct 30 21:45:01.680352 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:45:01.680324 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf00eb47b_3c81_4420_a948_45dae9800548.slice/crio-188a95ec77cb36a21021d3d84011140bf1a5d7f8278071fd886e7290dd0f23bd WatchSource:0}: Error finding container 188a95ec77cb36a21021d3d84011140bf1a5d7f8278071fd886e7290dd0f23bd: Status 404 returned error can't find the container with id 188a95ec77cb36a21021d3d84011140bf1a5d7f8278071fd886e7290dd0f23bd Oct 30 21:45:01.683821 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:45:01.683024 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42ef7e75_1cb4_4758_a7e3_5f6c3c6e7543.slice/crio-95b27353e9337829985abb9771d5d02a29c075a3e10a7986af9b43dedfb7d2a4 WatchSource:0}: Error finding container 95b27353e9337829985abb9771d5d02a29c075a3e10a7986af9b43dedfb7d2a4: Status 404 returned error can't find the container with id 95b27353e9337829985abb9771d5d02a29c075a3e10a7986af9b43dedfb7d2a4 Oct 30 21:45:01.689606 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:45:01.689572 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f27f31_e62c_482e_bfba_10c2250410a4.slice/crio-7075ae15bc1a57ebea482c6f2ae02879079f9a71e334199a03c0f37f0920b608 WatchSource:0}: Error finding container 7075ae15bc1a57ebea482c6f2ae02879079f9a71e334199a03c0f37f0920b608: Status 404 returned error can't find the container with id 7075ae15bc1a57ebea482c6f2ae02879079f9a71e334199a03c0f37f0920b608 Oct 30 21:45:01.792783 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:01.792744 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz"] Oct 30 21:45:02.020095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:02.019741 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t" event={"ID":"c61bf7ef-856d-4d2e-b4ca-402028fabd8f","Type":"ContainerStarted","Data":"a2f2b8f0ec6c13b1eb47d793c145dccb38735c688b1082262835ac443dc8c73e"} Oct 30 21:45:02.024651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:02.024621 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz" event={"ID":"e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3","Type":"ContainerStarted","Data":"876c39236f79f080758dc9671700c4481eb864da842b66764375a372de2914f6"} Oct 30 21:45:02.026265 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:02.026199 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm" event={"ID":"205a1101-2a99-4800-84d8-56a39d3690b6","Type":"ContainerStarted","Data":"97f47cf8554c07130b8b5625832b84abc723d9eabbe8effe552956fa9989b7c0"} Oct 30 21:45:02.026265 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:02.026243 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm" event={"ID":"205a1101-2a99-4800-84d8-56a39d3690b6","Type":"ContainerStarted","Data":"62dc66bcc454267c9480b53112e74494fad9e446dfe4adf9512cd03b4f4e4c9a"} Oct 30 21:45:02.027548 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:02.027525 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4" event={"ID":"30def38d-8a67-4281-9031-c6f7f7c96bc5","Type":"ContainerStarted","Data":"d54d5710d71fde2ba37668627a22cf8cb780578b64660b0f7ef8e8af69ef2a79"} Oct 30 21:45:02.028592 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:02.028570 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds" event={"ID":"596e496a-35f5-483f-bbde-62e3dc028766","Type":"ContainerStarted","Data":"a3f78e900443de35f32ee2a152ec52962f4a34212cbf2ea677b06b4dc2057d54"} Oct 30 21:45:02.029827 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:02.029619 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n" event={"ID":"b0bb7714-22f3-4fe2-887c-71d638b4d61d","Type":"ContainerStarted","Data":"c3fe9d7b605418a5029849a541abc1cb4708b075bd1dd1e68e6ff5534f34cfa4"} Oct 30 21:45:02.030666 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:02.030647 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k" event={"ID":"2a03d85d-cd11-4d9a-ae67-29ced159f159","Type":"ContainerStarted","Data":"34dd91abe0deb24548bf5f937062ba40f8cb35a98e9ff17754c0e67aa4fae8c0"} Oct 30 21:45:02.031682 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:02.031663 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj" event={"ID":"657d2744-b23d-4dbb-a75d-23fea5f9a286","Type":"ContainerStarted","Data":"399b43ed9c492f3f8f44f0687080aba39d1f1f192111d246f3a9a10134255bb7"} Oct 30 21:45:02.032687 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:02.032668 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p" event={"ID":"c51dea80-dea4-4195-8c0a-8f28ea265669","Type":"ContainerStarted","Data":"48e98b2025034bbc918471fd1615b9aa2aab9e759a39e2affe6de3e53cbb8de5"} Oct 30 21:45:02.033785 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:02.033766 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn" event={"ID":"19dafab6-6daa-4e31-bc89-3ebd04adffbb","Type":"ContainerStarted","Data":"08e0e754f285f6f329b04f7dd25542fb0cf7482c6d52d89e9053e093beb85413"} Oct 30 21:45:02.034819 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:02.034799 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz" event={"ID":"42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543","Type":"ContainerStarted","Data":"95b27353e9337829985abb9771d5d02a29c075a3e10a7986af9b43dedfb7d2a4"} Oct 30 21:45:02.035863 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:02.035844 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95" event={"ID":"bd055827-04e0-4946-bf79-b8e59e81fa21","Type":"ContainerStarted","Data":"6704aa357dcd1ed39c18c053dd76e3982bd195fd8480177cb2fb2f4f062fb533"} Oct 30 21:45:02.037963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:02.037942 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t" event={"ID":"f00eb47b-3c81-4420-a948-45dae9800548","Type":"ContainerStarted","Data":"188a95ec77cb36a21021d3d84011140bf1a5d7f8278071fd886e7290dd0f23bd"} Oct 30 21:45:02.038966 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:02.038944 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4" event={"ID":"d0f27f31-e62c-482e-bfba-10c2250410a4","Type":"ContainerStarted","Data":"7075ae15bc1a57ebea482c6f2ae02879079f9a71e334199a03c0f37f0920b608"} Oct 30 21:45:02.040667 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:02.040646 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2" event={"ID":"547cf37c-665f-4c4b-ab6f-06b5422b9cc9","Type":"ContainerStarted","Data":"4daabc084562e24487c798c2cecff51676d16268510139c1701622ef6a418d09"} Oct 30 21:45:02.040770 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:02.040681 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2" event={"ID":"547cf37c-665f-4c4b-ab6f-06b5422b9cc9","Type":"ContainerStarted","Data":"bb25c12a1a0337156145519c0f4c4a1e1ded9373017853e9d6ff25a9153b3ef6"} Oct 30 21:45:02.085263 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:02.084568 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm" podStartSLOduration=2.0845473820000002 podStartE2EDuration="2.084547382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.075467114 +0000 UTC m=+2689.196540853" watchObservedRunningTime="2026-10-30 21:45:02.084547382 +0000 UTC m=+2689.205621132" Oct 30 21:45:02.125793 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:02.125732 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2" podStartSLOduration=2.125717588 podStartE2EDuration="2.125717588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:02.125368337 +0000 UTC m=+2689.246442083" watchObservedRunningTime="2026-10-30 21:45:02.125717588 +0000 UTC m=+2689.246791334" Oct 30 21:45:03.057301 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.057265 4926 generic.go:356] "Generic (PLEG): container finished" podID="c61bf7ef-856d-4d2e-b4ca-402028fabd8f" containerID="4fa03848f43b99252ab88592d271142e17f8d624ba0e229e97584c6b679a6fb0" exitCode=60 Oct 30 21:45:03.057730 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.057364 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t" event={"ID":"c61bf7ef-856d-4d2e-b4ca-402028fabd8f","Type":"ContainerDied","Data":"4fa03848f43b99252ab88592d271142e17f8d624ba0e229e97584c6b679a6fb0"} Oct 30 21:45:03.057730 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.057677 4926 scope.go:122] "RemoveContainer" containerID="4fa03848f43b99252ab88592d271142e17f8d624ba0e229e97584c6b679a6fb0" Oct 30 21:45:03.059188 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.059160 4926 generic.go:356] "Generic (PLEG): container finished" podID="e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3" containerID="2d55595197c9c32cc188f915a7358944cee5512f118d9807272168bc423842fb" exitCode=60 Oct 30 21:45:03.059289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.059234 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz" event={"ID":"e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3","Type":"ContainerDied","Data":"2d55595197c9c32cc188f915a7358944cee5512f118d9807272168bc423842fb"} Oct 30 21:45:03.059500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.059469 4926 scope.go:122] "RemoveContainer" containerID="2d55595197c9c32cc188f915a7358944cee5512f118d9807272168bc423842fb" Oct 30 21:45:03.061262 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.061239 4926 generic.go:356] "Generic (PLEG): container finished" podID="205a1101-2a99-4800-84d8-56a39d3690b6" containerID="97f47cf8554c07130b8b5625832b84abc723d9eabbe8effe552956fa9989b7c0" exitCode=60 Oct 30 21:45:03.061347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.061307 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm" event={"ID":"205a1101-2a99-4800-84d8-56a39d3690b6","Type":"ContainerDied","Data":"97f47cf8554c07130b8b5625832b84abc723d9eabbe8effe552956fa9989b7c0"} Oct 30 21:45:03.061579 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.061560 4926 scope.go:122] "RemoveContainer" containerID="97f47cf8554c07130b8b5625832b84abc723d9eabbe8effe552956fa9989b7c0" Oct 30 21:45:03.063524 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.063504 4926 generic.go:356] "Generic (PLEG): container finished" podID="30def38d-8a67-4281-9031-c6f7f7c96bc5" containerID="a31cdf1cd17b2ffab6bd4654bec42e05e9274d136672af99bdf2fd2876cac369" exitCode=60 Oct 30 21:45:03.063617 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.063561 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4" event={"ID":"30def38d-8a67-4281-9031-c6f7f7c96bc5","Type":"ContainerDied","Data":"a31cdf1cd17b2ffab6bd4654bec42e05e9274d136672af99bdf2fd2876cac369"} Oct 30 21:45:03.063800 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.063783 4926 scope.go:122] "RemoveContainer" containerID="a31cdf1cd17b2ffab6bd4654bec42e05e9274d136672af99bdf2fd2876cac369" Oct 30 21:45:03.065279 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.065259 4926 generic.go:356] "Generic (PLEG): container finished" podID="596e496a-35f5-483f-bbde-62e3dc028766" containerID="da73caf10c0b49e18e34f7533704297c7ba1dd4ead40d423a0df6860ef6da9a0" exitCode=60 Oct 30 21:45:03.065371 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.065337 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds" event={"ID":"596e496a-35f5-483f-bbde-62e3dc028766","Type":"ContainerDied","Data":"da73caf10c0b49e18e34f7533704297c7ba1dd4ead40d423a0df6860ef6da9a0"} Oct 30 21:45:03.065529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.065513 4926 scope.go:122] "RemoveContainer" containerID="da73caf10c0b49e18e34f7533704297c7ba1dd4ead40d423a0df6860ef6da9a0" Oct 30 21:45:03.067243 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.067220 4926 generic.go:356] "Generic (PLEG): container finished" podID="b0bb7714-22f3-4fe2-887c-71d638b4d61d" containerID="98a7b972995fd72ccd5533dc8c06787dc175c1c6dcb7c0dd621a04b4d4a2fbf1" exitCode=60 Oct 30 21:45:03.067342 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.067302 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n" event={"ID":"b0bb7714-22f3-4fe2-887c-71d638b4d61d","Type":"ContainerDied","Data":"98a7b972995fd72ccd5533dc8c06787dc175c1c6dcb7c0dd621a04b4d4a2fbf1"} Oct 30 21:45:03.067529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.067512 4926 scope.go:122] "RemoveContainer" containerID="98a7b972995fd72ccd5533dc8c06787dc175c1c6dcb7c0dd621a04b4d4a2fbf1" Oct 30 21:45:03.069300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.069279 4926 generic.go:356] "Generic (PLEG): container finished" podID="2a03d85d-cd11-4d9a-ae67-29ced159f159" containerID="6f5a8b071bafa6f4bb835b3f1939700dec621ffae4d59898052f9783dd64f883" exitCode=60 Oct 30 21:45:03.069399 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.069328 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k" event={"ID":"2a03d85d-cd11-4d9a-ae67-29ced159f159","Type":"ContainerDied","Data":"6f5a8b071bafa6f4bb835b3f1939700dec621ffae4d59898052f9783dd64f883"} Oct 30 21:45:03.069604 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.069586 4926 scope.go:122] "RemoveContainer" containerID="6f5a8b071bafa6f4bb835b3f1939700dec621ffae4d59898052f9783dd64f883" Oct 30 21:45:03.075610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.075588 4926 generic.go:356] "Generic (PLEG): container finished" podID="657d2744-b23d-4dbb-a75d-23fea5f9a286" containerID="e47025d5f009b3c3742dac066ca084c62a1b93943ae52244be39630ae86a8486" exitCode=60 Oct 30 21:45:03.075731 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.075663 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj" event={"ID":"657d2744-b23d-4dbb-a75d-23fea5f9a286","Type":"ContainerDied","Data":"e47025d5f009b3c3742dac066ca084c62a1b93943ae52244be39630ae86a8486"} Oct 30 21:45:03.075954 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.075938 4926 scope.go:122] "RemoveContainer" containerID="e47025d5f009b3c3742dac066ca084c62a1b93943ae52244be39630ae86a8486" Oct 30 21:45:03.079284 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.079258 4926 generic.go:356] "Generic (PLEG): container finished" podID="c51dea80-dea4-4195-8c0a-8f28ea265669" containerID="2f5659059e8736a5a5a5afec20ba80e411dd90c37ee13405d5d0d19fe73c85bd" exitCode=60 Oct 30 21:45:03.079406 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.079329 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p" event={"ID":"c51dea80-dea4-4195-8c0a-8f28ea265669","Type":"ContainerDied","Data":"2f5659059e8736a5a5a5afec20ba80e411dd90c37ee13405d5d0d19fe73c85bd"} Oct 30 21:45:03.079658 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.079639 4926 scope.go:122] "RemoveContainer" containerID="2f5659059e8736a5a5a5afec20ba80e411dd90c37ee13405d5d0d19fe73c85bd" Oct 30 21:45:03.084426 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.084404 4926 generic.go:356] "Generic (PLEG): container finished" podID="19dafab6-6daa-4e31-bc89-3ebd04adffbb" containerID="9513d5ce0d030f528e4f57a696f31ecfe7ab7bc20d4baafceff0b9b03b540fd6" exitCode=60 Oct 30 21:45:03.084628 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.084610 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn" event={"ID":"19dafab6-6daa-4e31-bc89-3ebd04adffbb","Type":"ContainerDied","Data":"9513d5ce0d030f528e4f57a696f31ecfe7ab7bc20d4baafceff0b9b03b540fd6"} Oct 30 21:45:03.085008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.084989 4926 scope.go:122] "RemoveContainer" containerID="9513d5ce0d030f528e4f57a696f31ecfe7ab7bc20d4baafceff0b9b03b540fd6" Oct 30 21:45:03.089417 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.089298 4926 generic.go:356] "Generic (PLEG): container finished" podID="42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543" containerID="ab1eda69c3258f2903a475888e873a6391ed218091860149e9eff2a911b494ce" exitCode=60 Oct 30 21:45:03.089417 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.089365 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz" event={"ID":"42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543","Type":"ContainerDied","Data":"ab1eda69c3258f2903a475888e873a6391ed218091860149e9eff2a911b494ce"} Oct 30 21:45:03.089588 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.089572 4926 scope.go:122] "RemoveContainer" containerID="ab1eda69c3258f2903a475888e873a6391ed218091860149e9eff2a911b494ce" Oct 30 21:45:03.091865 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.091841 4926 generic.go:356] "Generic (PLEG): container finished" podID="bd055827-04e0-4946-bf79-b8e59e81fa21" containerID="3166798afe2cd5258aea0c25aa157f67e4beccef3c2dd2c32e97ce1675a16cb0" exitCode=60 Oct 30 21:45:03.092094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.091992 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95" event={"ID":"bd055827-04e0-4946-bf79-b8e59e81fa21","Type":"ContainerDied","Data":"3166798afe2cd5258aea0c25aa157f67e4beccef3c2dd2c32e97ce1675a16cb0"} Oct 30 21:45:03.092197 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.092140 4926 scope.go:122] "RemoveContainer" containerID="3166798afe2cd5258aea0c25aa157f67e4beccef3c2dd2c32e97ce1675a16cb0" Oct 30 21:45:03.094173 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.094147 4926 generic.go:356] "Generic (PLEG): container finished" podID="f00eb47b-3c81-4420-a948-45dae9800548" containerID="0ab0a0349169ad8ad6fd0e1da604266a74749a3074a6f03c2af975efbdbeff5d" exitCode=60 Oct 30 21:45:03.094276 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.094225 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t" event={"ID":"f00eb47b-3c81-4420-a948-45dae9800548","Type":"ContainerDied","Data":"0ab0a0349169ad8ad6fd0e1da604266a74749a3074a6f03c2af975efbdbeff5d"} Oct 30 21:45:03.094505 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.094485 4926 scope.go:122] "RemoveContainer" containerID="0ab0a0349169ad8ad6fd0e1da604266a74749a3074a6f03c2af975efbdbeff5d" Oct 30 21:45:03.101620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.100691 4926 generic.go:356] "Generic (PLEG): container finished" podID="d0f27f31-e62c-482e-bfba-10c2250410a4" containerID="afe587f72808ef0854c24b4aa22a9681e5d95d9cfa942c4c65e445acebed87ea" exitCode=60 Oct 30 21:45:03.101620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.100777 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4" event={"ID":"d0f27f31-e62c-482e-bfba-10c2250410a4","Type":"ContainerDied","Data":"afe587f72808ef0854c24b4aa22a9681e5d95d9cfa942c4c65e445acebed87ea"} Oct 30 21:45:03.101620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.101161 4926 scope.go:122] "RemoveContainer" containerID="afe587f72808ef0854c24b4aa22a9681e5d95d9cfa942c4c65e445acebed87ea" Oct 30 21:45:03.109217 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.109190 4926 generic.go:356] "Generic (PLEG): container finished" podID="547cf37c-665f-4c4b-ab6f-06b5422b9cc9" containerID="4daabc084562e24487c798c2cecff51676d16268510139c1701622ef6a418d09" exitCode=60 Oct 30 21:45:03.109327 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.109259 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2" event={"ID":"547cf37c-665f-4c4b-ab6f-06b5422b9cc9","Type":"ContainerDied","Data":"4daabc084562e24487c798c2cecff51676d16268510139c1701622ef6a418d09"} Oct 30 21:45:03.109530 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:03.109511 4926 scope.go:122] "RemoveContainer" containerID="4daabc084562e24487c798c2cecff51676d16268510139c1701622ef6a418d09" Oct 30 21:45:04.120489 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:04.120436 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n" event={"ID":"b0bb7714-22f3-4fe2-887c-71d638b4d61d","Type":"ContainerStarted","Data":"e29e96aca2e4c77f84a463b63f0856543325584f26014a52467ca1e041ceac9b"} Oct 30 21:45:04.122892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:04.122853 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p" event={"ID":"c51dea80-dea4-4195-8c0a-8f28ea265669","Type":"ContainerStarted","Data":"78f35822e24ae935edbfdafa47a0e564f527b4ada1e5cccc8e899e0d3f91b5a8"} Oct 30 21:45:04.125191 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:04.125154 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn" event={"ID":"19dafab6-6daa-4e31-bc89-3ebd04adffbb","Type":"ContainerStarted","Data":"f735de216a256ec53002002645601c5a4ece70d8e4156703fd842a483a830020"} Oct 30 21:45:04.127487 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:04.127456 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t" event={"ID":"f00eb47b-3c81-4420-a948-45dae9800548","Type":"ContainerStarted","Data":"2173966983d96371216428a33372bd3de88436c396279849db4548ab16b5c80a"} Oct 30 21:45:04.129626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:04.129584 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2" event={"ID":"547cf37c-665f-4c4b-ab6f-06b5422b9cc9","Type":"ContainerStarted","Data":"76ea305952dc23ebc2bb8988d1bf45e9b520839b3056068112e077e266273ec5"} Oct 30 21:45:04.131893 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:04.131866 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t" event={"ID":"c61bf7ef-856d-4d2e-b4ca-402028fabd8f","Type":"ContainerStarted","Data":"1eb6abaafcbc093b287fe49f396d1612e62007d294774d4ec338f6fc0134f186"} Oct 30 21:45:04.180007 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:04.179939 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n" podStartSLOduration=4.179917797 podStartE2EDuration="4.179917797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.17355001 +0000 UTC m=+2691.294623757" watchObservedRunningTime="2026-10-30 21:45:04.179917797 +0000 UTC m=+2691.300991544" Oct 30 21:45:04.280897 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:04.279882 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t" podStartSLOduration=4.279862488 podStartE2EDuration="4.279862488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.277106344 +0000 UTC m=+2691.398180092" watchObservedRunningTime="2026-10-30 21:45:04.279862488 +0000 UTC m=+2691.400936235" Oct 30 21:45:04.322163 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:04.322115 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p" podStartSLOduration=4.322099991 podStartE2EDuration="4.322099991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.320249723 +0000 UTC m=+2691.441323468" watchObservedRunningTime="2026-10-30 21:45:04.322099991 +0000 UTC m=+2691.443173736" Oct 30 21:45:04.363560 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:04.363450 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t" podStartSLOduration=4.363435504 podStartE2EDuration="4.363435504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.361603496 +0000 UTC m=+2691.482677241" watchObservedRunningTime="2026-10-30 21:45:04.363435504 +0000 UTC m=+2691.484509249" Oct 30 21:45:04.406372 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:04.406301 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn" podStartSLOduration=4.406283899 podStartE2EDuration="4.406283899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:04.404649727 +0000 UTC m=+2691.525723474" watchObservedRunningTime="2026-10-30 21:45:04.406283899 +0000 UTC m=+2691.527357646" Oct 30 21:45:05.142940 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.142908 4926 generic.go:356] "Generic (PLEG): container finished" podID="c61bf7ef-856d-4d2e-b4ca-402028fabd8f" containerID="1eb6abaafcbc093b287fe49f396d1612e62007d294774d4ec338f6fc0134f186" exitCode=60 Oct 30 21:45:05.143365 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.142993 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t" event={"ID":"c61bf7ef-856d-4d2e-b4ca-402028fabd8f","Type":"ContainerDied","Data":"1eb6abaafcbc093b287fe49f396d1612e62007d294774d4ec338f6fc0134f186"} Oct 30 21:45:05.143365 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.143114 4926 scope.go:122] "RemoveContainer" containerID="4fa03848f43b99252ab88592d271142e17f8d624ba0e229e97584c6b679a6fb0" Oct 30 21:45:05.143365 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.143349 4926 scope.go:122] "RemoveContainer" containerID="1eb6abaafcbc093b287fe49f396d1612e62007d294774d4ec338f6fc0134f186" Oct 30 21:45:05.143762 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:05.143735 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t_odh-ai-gateway-infra(c61bf7ef-856d-4d2e-b4ca-402028fabd8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t" podUID="c61bf7ef-856d-4d2e-b4ca-402028fabd8f" Oct 30 21:45:05.145094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.145074 4926 generic.go:356] "Generic (PLEG): container finished" podID="e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3" containerID="a20ef0ccb0dfbdeed8eabe7d824b1cb32d27953c03ea4e289aa2fe00cb2aec15" exitCode=60 Oct 30 21:45:05.145199 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.145153 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz" event={"ID":"e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3","Type":"ContainerDied","Data":"a20ef0ccb0dfbdeed8eabe7d824b1cb32d27953c03ea4e289aa2fe00cb2aec15"} Oct 30 21:45:05.145502 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.145487 4926 scope.go:122] "RemoveContainer" containerID="a20ef0ccb0dfbdeed8eabe7d824b1cb32d27953c03ea4e289aa2fe00cb2aec15" Oct 30 21:45:05.145830 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:05.145806 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz_odh-ai-gateway-infra(e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz" podUID="e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3" Oct 30 21:45:05.147252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.147228 4926 generic.go:356] "Generic (PLEG): container finished" podID="205a1101-2a99-4800-84d8-56a39d3690b6" containerID="42b7a51cafd8c81856679c474bd70cea9e5073888226dc5c73d0b85639212621" exitCode=60 Oct 30 21:45:05.147356 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.147314 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm" event={"ID":"205a1101-2a99-4800-84d8-56a39d3690b6","Type":"ContainerDied","Data":"42b7a51cafd8c81856679c474bd70cea9e5073888226dc5c73d0b85639212621"} Oct 30 21:45:05.147630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.147613 4926 scope.go:122] "RemoveContainer" containerID="42b7a51cafd8c81856679c474bd70cea9e5073888226dc5c73d0b85639212621" Oct 30 21:45:05.147956 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:05.147937 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm_odh-ai-gateway-infra(205a1101-2a99-4800-84d8-56a39d3690b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm" podUID="205a1101-2a99-4800-84d8-56a39d3690b6" Oct 30 21:45:05.149344 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.149324 4926 generic.go:356] "Generic (PLEG): container finished" podID="30def38d-8a67-4281-9031-c6f7f7c96bc5" containerID="641a45476084c85176ebe060d1d723510f7c8ffb4506520b23abaffc3e4f4177" exitCode=60 Oct 30 21:45:05.149443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.149378 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4" event={"ID":"30def38d-8a67-4281-9031-c6f7f7c96bc5","Type":"ContainerDied","Data":"641a45476084c85176ebe060d1d723510f7c8ffb4506520b23abaffc3e4f4177"} Oct 30 21:45:05.149664 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.149648 4926 scope.go:122] "RemoveContainer" containerID="641a45476084c85176ebe060d1d723510f7c8ffb4506520b23abaffc3e4f4177" Oct 30 21:45:05.149993 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:05.149972 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4_odh-ai-gateway-infra(30def38d-8a67-4281-9031-c6f7f7c96bc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4" podUID="30def38d-8a67-4281-9031-c6f7f7c96bc5" Oct 30 21:45:05.151409 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.151386 4926 generic.go:356] "Generic (PLEG): container finished" podID="596e496a-35f5-483f-bbde-62e3dc028766" containerID="3b737135da42794cbea19f17d00efd10b4110c6a9a6ee87e66f26e95621540c6" exitCode=60 Oct 30 21:45:05.151487 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.151467 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds" event={"ID":"596e496a-35f5-483f-bbde-62e3dc028766","Type":"ContainerDied","Data":"3b737135da42794cbea19f17d00efd10b4110c6a9a6ee87e66f26e95621540c6"} Oct 30 21:45:05.151702 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.151687 4926 scope.go:122] "RemoveContainer" containerID="3b737135da42794cbea19f17d00efd10b4110c6a9a6ee87e66f26e95621540c6" Oct 30 21:45:05.151954 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:05.151931 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds_odh-ai-gateway-infra(596e496a-35f5-483f-bbde-62e3dc028766)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds" podUID="596e496a-35f5-483f-bbde-62e3dc028766" Oct 30 21:45:05.153415 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.153394 4926 generic.go:356] "Generic (PLEG): container finished" podID="b0bb7714-22f3-4fe2-887c-71d638b4d61d" containerID="e29e96aca2e4c77f84a463b63f0856543325584f26014a52467ca1e041ceac9b" exitCode=60 Oct 30 21:45:05.153487 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.153468 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n" event={"ID":"b0bb7714-22f3-4fe2-887c-71d638b4d61d","Type":"ContainerDied","Data":"e29e96aca2e4c77f84a463b63f0856543325584f26014a52467ca1e041ceac9b"} Oct 30 21:45:05.153691 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.153676 4926 scope.go:122] "RemoveContainer" containerID="e29e96aca2e4c77f84a463b63f0856543325584f26014a52467ca1e041ceac9b" Oct 30 21:45:05.153938 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:05.153916 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n_odh-ai-gateway-infra(b0bb7714-22f3-4fe2-887c-71d638b4d61d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n" podUID="b0bb7714-22f3-4fe2-887c-71d638b4d61d" Oct 30 21:45:05.155421 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.155395 4926 generic.go:356] "Generic (PLEG): container finished" podID="2a03d85d-cd11-4d9a-ae67-29ced159f159" containerID="0c6d61d16b7593cd1fa651ce2442f362642b69ea6cda338470b7897a852a6e91" exitCode=60 Oct 30 21:45:05.155508 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.155458 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k" event={"ID":"2a03d85d-cd11-4d9a-ae67-29ced159f159","Type":"ContainerDied","Data":"0c6d61d16b7593cd1fa651ce2442f362642b69ea6cda338470b7897a852a6e91"} Oct 30 21:45:05.155787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.155751 4926 scope.go:122] "RemoveContainer" containerID="0c6d61d16b7593cd1fa651ce2442f362642b69ea6cda338470b7897a852a6e91" Oct 30 21:45:05.156248 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:05.156225 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k_odh-ai-gateway-infra(2a03d85d-cd11-4d9a-ae67-29ced159f159)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k" podUID="2a03d85d-cd11-4d9a-ae67-29ced159f159" Oct 30 21:45:05.157617 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.157596 4926 generic.go:356] "Generic (PLEG): container finished" podID="657d2744-b23d-4dbb-a75d-23fea5f9a286" containerID="19e35d824f0b11b5caf3db896ca0c2a72917d786c6b891a3a19befa71a1c705f" exitCode=60 Oct 30 21:45:05.157709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.157686 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj" event={"ID":"657d2744-b23d-4dbb-a75d-23fea5f9a286","Type":"ContainerDied","Data":"19e35d824f0b11b5caf3db896ca0c2a72917d786c6b891a3a19befa71a1c705f"} Oct 30 21:45:05.157904 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.157890 4926 scope.go:122] "RemoveContainer" containerID="19e35d824f0b11b5caf3db896ca0c2a72917d786c6b891a3a19befa71a1c705f" Oct 30 21:45:05.158165 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:05.158142 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj_odh-ai-gateway-infra(657d2744-b23d-4dbb-a75d-23fea5f9a286)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj" podUID="657d2744-b23d-4dbb-a75d-23fea5f9a286" Oct 30 21:45:05.159553 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.159535 4926 generic.go:356] "Generic (PLEG): container finished" podID="c51dea80-dea4-4195-8c0a-8f28ea265669" containerID="78f35822e24ae935edbfdafa47a0e564f527b4ada1e5cccc8e899e0d3f91b5a8" exitCode=60 Oct 30 21:45:05.159663 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.159598 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p" event={"ID":"c51dea80-dea4-4195-8c0a-8f28ea265669","Type":"ContainerDied","Data":"78f35822e24ae935edbfdafa47a0e564f527b4ada1e5cccc8e899e0d3f91b5a8"} Oct 30 21:45:05.159805 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.159788 4926 scope.go:122] "RemoveContainer" containerID="78f35822e24ae935edbfdafa47a0e564f527b4ada1e5cccc8e899e0d3f91b5a8" Oct 30 21:45:05.160048 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:05.160013 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p_odh-ai-gateway-infra(c51dea80-dea4-4195-8c0a-8f28ea265669)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p" podUID="c51dea80-dea4-4195-8c0a-8f28ea265669" Oct 30 21:45:05.161464 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.161443 4926 generic.go:356] "Generic (PLEG): container finished" podID="19dafab6-6daa-4e31-bc89-3ebd04adffbb" containerID="f735de216a256ec53002002645601c5a4ece70d8e4156703fd842a483a830020" exitCode=60 Oct 30 21:45:05.161538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.161510 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn" event={"ID":"19dafab6-6daa-4e31-bc89-3ebd04adffbb","Type":"ContainerDied","Data":"f735de216a256ec53002002645601c5a4ece70d8e4156703fd842a483a830020"} Oct 30 21:45:05.161700 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.161683 4926 scope.go:122] "RemoveContainer" containerID="f735de216a256ec53002002645601c5a4ece70d8e4156703fd842a483a830020" Oct 30 21:45:05.161916 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:05.161897 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn_odh-ai-gateway-infra(19dafab6-6daa-4e31-bc89-3ebd04adffbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn" podUID="19dafab6-6daa-4e31-bc89-3ebd04adffbb" Oct 30 21:45:05.163418 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.163400 4926 generic.go:356] "Generic (PLEG): container finished" podID="42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543" containerID="918ad2c9e7b0c1fa9fe04748f27d0c432841e6d5812fa9e39ceb8531a39de71c" exitCode=60 Oct 30 21:45:05.163506 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.163466 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz" event={"ID":"42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543","Type":"ContainerDied","Data":"918ad2c9e7b0c1fa9fe04748f27d0c432841e6d5812fa9e39ceb8531a39de71c"} Oct 30 21:45:05.163699 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.163681 4926 scope.go:122] "RemoveContainer" containerID="918ad2c9e7b0c1fa9fe04748f27d0c432841e6d5812fa9e39ceb8531a39de71c" Oct 30 21:45:05.163921 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:05.163904 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz_odh-ai-gateway-infra(42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz" podUID="42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543" Oct 30 21:45:05.165703 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.165679 4926 generic.go:356] "Generic (PLEG): container finished" podID="bd055827-04e0-4946-bf79-b8e59e81fa21" containerID="6f4b3e32e7c710b63e4e97b756538732f295281196d6caeeb7eaf0456549f0ed" exitCode=60 Oct 30 21:45:05.165811 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.165785 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95" event={"ID":"bd055827-04e0-4946-bf79-b8e59e81fa21","Type":"ContainerDied","Data":"6f4b3e32e7c710b63e4e97b756538732f295281196d6caeeb7eaf0456549f0ed"} Oct 30 21:45:05.166070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.166050 4926 scope.go:122] "RemoveContainer" containerID="6f4b3e32e7c710b63e4e97b756538732f295281196d6caeeb7eaf0456549f0ed" Oct 30 21:45:05.166298 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:05.166279 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-5sb95_odh-ai-gateway-infra(bd055827-04e0-4946-bf79-b8e59e81fa21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95" podUID="bd055827-04e0-4946-bf79-b8e59e81fa21" Oct 30 21:45:05.168014 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.167991 4926 generic.go:356] "Generic (PLEG): container finished" podID="f00eb47b-3c81-4420-a948-45dae9800548" containerID="2173966983d96371216428a33372bd3de88436c396279849db4548ab16b5c80a" exitCode=60 Oct 30 21:45:05.168144 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.168018 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t" event={"ID":"f00eb47b-3c81-4420-a948-45dae9800548","Type":"ContainerDied","Data":"2173966983d96371216428a33372bd3de88436c396279849db4548ab16b5c80a"} Oct 30 21:45:05.168330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.168289 4926 scope.go:122] "RemoveContainer" containerID="2173966983d96371216428a33372bd3de88436c396279849db4548ab16b5c80a" Oct 30 21:45:05.168556 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:05.168530 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t_odh-ai-gateway-infra(f00eb47b-3c81-4420-a948-45dae9800548)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t" podUID="f00eb47b-3c81-4420-a948-45dae9800548" Oct 30 21:45:05.170366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.170344 4926 generic.go:356] "Generic (PLEG): container finished" podID="d0f27f31-e62c-482e-bfba-10c2250410a4" containerID="0e112c22d0d6cd6e8ab8cff8fe0f8d4562c9cfbbf11bb0590612fa394391dcc7" exitCode=60 Oct 30 21:45:05.170474 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.170414 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4" event={"ID":"d0f27f31-e62c-482e-bfba-10c2250410a4","Type":"ContainerDied","Data":"0e112c22d0d6cd6e8ab8cff8fe0f8d4562c9cfbbf11bb0590612fa394391dcc7"} Oct 30 21:45:05.170682 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.170658 4926 scope.go:122] "RemoveContainer" containerID="0e112c22d0d6cd6e8ab8cff8fe0f8d4562c9cfbbf11bb0590612fa394391dcc7" Oct 30 21:45:05.170919 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:05.170898 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4_odh-ai-gateway-infra(d0f27f31-e62c-482e-bfba-10c2250410a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4" podUID="d0f27f31-e62c-482e-bfba-10c2250410a4" Oct 30 21:45:05.172769 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.172745 4926 generic.go:356] "Generic (PLEG): container finished" podID="547cf37c-665f-4c4b-ab6f-06b5422b9cc9" containerID="76ea305952dc23ebc2bb8988d1bf45e9b520839b3056068112e077e266273ec5" exitCode=60 Oct 30 21:45:05.172847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.172779 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2" event={"ID":"547cf37c-665f-4c4b-ab6f-06b5422b9cc9","Type":"ContainerDied","Data":"76ea305952dc23ebc2bb8988d1bf45e9b520839b3056068112e077e266273ec5"} Oct 30 21:45:05.173094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.173077 4926 scope.go:122] "RemoveContainer" containerID="76ea305952dc23ebc2bb8988d1bf45e9b520839b3056068112e077e266273ec5" Oct 30 21:45:05.173344 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:05.173325 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2_odh-ai-gateway-infra(547cf37c-665f-4c4b-ab6f-06b5422b9cc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2" podUID="547cf37c-665f-4c4b-ab6f-06b5422b9cc9" Oct 30 21:45:05.486120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.486089 4926 scope.go:122] "RemoveContainer" containerID="2d55595197c9c32cc188f915a7358944cee5512f118d9807272168bc423842fb" Oct 30 21:45:05.504744 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.504716 4926 scope.go:122] "RemoveContainer" containerID="97f47cf8554c07130b8b5625832b84abc723d9eabbe8effe552956fa9989b7c0" Oct 30 21:45:05.521705 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.521680 4926 scope.go:122] "RemoveContainer" containerID="a31cdf1cd17b2ffab6bd4654bec42e05e9274d136672af99bdf2fd2876cac369" Oct 30 21:45:05.536621 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.536585 4926 scope.go:122] "RemoveContainer" containerID="da73caf10c0b49e18e34f7533704297c7ba1dd4ead40d423a0df6860ef6da9a0" Oct 30 21:45:05.554022 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.553994 4926 scope.go:122] "RemoveContainer" containerID="98a7b972995fd72ccd5533dc8c06787dc175c1c6dcb7c0dd621a04b4d4a2fbf1" Oct 30 21:45:05.576565 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.576532 4926 scope.go:122] "RemoveContainer" containerID="6f5a8b071bafa6f4bb835b3f1939700dec621ffae4d59898052f9783dd64f883" Oct 30 21:45:05.595737 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.595714 4926 scope.go:122] "RemoveContainer" containerID="e47025d5f009b3c3742dac066ca084c62a1b93943ae52244be39630ae86a8486" Oct 30 21:45:05.612521 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.612494 4926 scope.go:122] "RemoveContainer" containerID="2f5659059e8736a5a5a5afec20ba80e411dd90c37ee13405d5d0d19fe73c85bd" Oct 30 21:45:05.628215 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.628188 4926 scope.go:122] "RemoveContainer" containerID="9513d5ce0d030f528e4f57a696f31ecfe7ab7bc20d4baafceff0b9b03b540fd6" Oct 30 21:45:05.643103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.643072 4926 scope.go:122] "RemoveContainer" containerID="ab1eda69c3258f2903a475888e873a6391ed218091860149e9eff2a911b494ce" Oct 30 21:45:05.657843 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.657815 4926 scope.go:122] "RemoveContainer" containerID="3166798afe2cd5258aea0c25aa157f67e4beccef3c2dd2c32e97ce1675a16cb0" Oct 30 21:45:05.675251 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.675219 4926 scope.go:122] "RemoveContainer" containerID="0ab0a0349169ad8ad6fd0e1da604266a74749a3074a6f03c2af975efbdbeff5d" Oct 30 21:45:05.706296 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.706270 4926 scope.go:122] "RemoveContainer" containerID="afe587f72808ef0854c24b4aa22a9681e5d95d9cfa942c4c65e445acebed87ea" Oct 30 21:45:05.731320 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:05.731286 4926 scope.go:122] "RemoveContainer" containerID="4daabc084562e24487c798c2cecff51676d16268510139c1701622ef6a418d09" Oct 30 21:45:06.185625 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:06.185590 4926 scope.go:122] "RemoveContainer" containerID="1eb6abaafcbc093b287fe49f396d1612e62007d294774d4ec338f6fc0134f186" Oct 30 21:45:06.186108 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:06.186004 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t_odh-ai-gateway-infra(c61bf7ef-856d-4d2e-b4ca-402028fabd8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t" podUID="c61bf7ef-856d-4d2e-b4ca-402028fabd8f" Oct 30 21:45:06.187519 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:06.187494 4926 scope.go:122] "RemoveContainer" containerID="a20ef0ccb0dfbdeed8eabe7d824b1cb32d27953c03ea4e289aa2fe00cb2aec15" Oct 30 21:45:06.187861 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:06.187834 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz_odh-ai-gateway-infra(e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz" podUID="e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3" Oct 30 21:45:06.189538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:06.189513 4926 scope.go:122] "RemoveContainer" containerID="42b7a51cafd8c81856679c474bd70cea9e5073888226dc5c73d0b85639212621" Oct 30 21:45:06.189898 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:06.189859 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm_odh-ai-gateway-infra(205a1101-2a99-4800-84d8-56a39d3690b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm" podUID="205a1101-2a99-4800-84d8-56a39d3690b6" Oct 30 21:45:06.191222 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:06.191201 4926 scope.go:122] "RemoveContainer" containerID="641a45476084c85176ebe060d1d723510f7c8ffb4506520b23abaffc3e4f4177" Oct 30 21:45:06.191500 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:06.191475 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4_odh-ai-gateway-infra(30def38d-8a67-4281-9031-c6f7f7c96bc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4" podUID="30def38d-8a67-4281-9031-c6f7f7c96bc5" Oct 30 21:45:06.192997 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:06.192969 4926 scope.go:122] "RemoveContainer" containerID="3b737135da42794cbea19f17d00efd10b4110c6a9a6ee87e66f26e95621540c6" Oct 30 21:45:06.193309 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:06.193280 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds_odh-ai-gateway-infra(596e496a-35f5-483f-bbde-62e3dc028766)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds" podUID="596e496a-35f5-483f-bbde-62e3dc028766" Oct 30 21:45:06.194949 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:06.194927 4926 scope.go:122] "RemoveContainer" containerID="e29e96aca2e4c77f84a463b63f0856543325584f26014a52467ca1e041ceac9b" Oct 30 21:45:06.195260 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:06.195236 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n_odh-ai-gateway-infra(b0bb7714-22f3-4fe2-887c-71d638b4d61d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n" podUID="b0bb7714-22f3-4fe2-887c-71d638b4d61d" Oct 30 21:45:06.196811 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:06.196793 4926 scope.go:122] "RemoveContainer" containerID="0c6d61d16b7593cd1fa651ce2442f362642b69ea6cda338470b7897a852a6e91" Oct 30 21:45:06.197101 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:06.197074 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k_odh-ai-gateway-infra(2a03d85d-cd11-4d9a-ae67-29ced159f159)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k" podUID="2a03d85d-cd11-4d9a-ae67-29ced159f159" Oct 30 21:45:06.198631 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:06.198612 4926 scope.go:122] "RemoveContainer" containerID="19e35d824f0b11b5caf3db896ca0c2a72917d786c6b891a3a19befa71a1c705f" Oct 30 21:45:06.198889 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:06.198865 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj_odh-ai-gateway-infra(657d2744-b23d-4dbb-a75d-23fea5f9a286)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj" podUID="657d2744-b23d-4dbb-a75d-23fea5f9a286" Oct 30 21:45:06.200436 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:06.200414 4926 scope.go:122] "RemoveContainer" containerID="78f35822e24ae935edbfdafa47a0e564f527b4ada1e5cccc8e899e0d3f91b5a8" Oct 30 21:45:06.200653 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:06.200628 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p_odh-ai-gateway-infra(c51dea80-dea4-4195-8c0a-8f28ea265669)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p" podUID="c51dea80-dea4-4195-8c0a-8f28ea265669" Oct 30 21:45:06.202279 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:06.202260 4926 scope.go:122] "RemoveContainer" containerID="f735de216a256ec53002002645601c5a4ece70d8e4156703fd842a483a830020" Oct 30 21:45:06.202455 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:06.202437 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn_odh-ai-gateway-infra(19dafab6-6daa-4e31-bc89-3ebd04adffbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn" podUID="19dafab6-6daa-4e31-bc89-3ebd04adffbb" Oct 30 21:45:06.204094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:06.204075 4926 scope.go:122] "RemoveContainer" containerID="918ad2c9e7b0c1fa9fe04748f27d0c432841e6d5812fa9e39ceb8531a39de71c" Oct 30 21:45:06.204312 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:06.204292 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz_odh-ai-gateway-infra(42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz" podUID="42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543" Oct 30 21:45:06.205952 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:06.205926 4926 scope.go:122] "RemoveContainer" containerID="6f4b3e32e7c710b63e4e97b756538732f295281196d6caeeb7eaf0456549f0ed" Oct 30 21:45:06.206199 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:06.206173 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-5sb95_odh-ai-gateway-infra(bd055827-04e0-4946-bf79-b8e59e81fa21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95" podUID="bd055827-04e0-4946-bf79-b8e59e81fa21" Oct 30 21:45:06.207649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:06.207628 4926 scope.go:122] "RemoveContainer" containerID="2173966983d96371216428a33372bd3de88436c396279849db4548ab16b5c80a" Oct 30 21:45:06.207873 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:06.207851 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t_odh-ai-gateway-infra(f00eb47b-3c81-4420-a948-45dae9800548)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t" podUID="f00eb47b-3c81-4420-a948-45dae9800548" Oct 30 21:45:06.209281 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:06.209262 4926 scope.go:122] "RemoveContainer" containerID="0e112c22d0d6cd6e8ab8cff8fe0f8d4562c9cfbbf11bb0590612fa394391dcc7" Oct 30 21:45:06.209499 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:06.209477 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4_odh-ai-gateway-infra(d0f27f31-e62c-482e-bfba-10c2250410a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4" podUID="d0f27f31-e62c-482e-bfba-10c2250410a4" Oct 30 21:45:06.210956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:06.210940 4926 scope.go:122] "RemoveContainer" containerID="76ea305952dc23ebc2bb8988d1bf45e9b520839b3056068112e077e266273ec5" Oct 30 21:45:06.211180 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:06.211159 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2_odh-ai-gateway-infra(547cf37c-665f-4c4b-ab6f-06b5422b9cc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2" podUID="547cf37c-665f-4c4b-ab6f-06b5422b9cc9" Oct 30 21:45:07.495616 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:07.495569 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:45:11.018075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:11.018003 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt" Oct 30 21:45:12.045196 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:12.045162 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7" Oct 30 21:45:13.699477 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:13.699449 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:45:13.699477 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:13.699468 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:45:15.009502 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:15.009461 4926 scope.go:122] "RemoveContainer" containerID="3b737135da42794cbea19f17d00efd10b4110c6a9a6ee87e66f26e95621540c6" Oct 30 21:45:16.009424 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:16.009374 4926 scope.go:122] "RemoveContainer" containerID="78f35822e24ae935edbfdafa47a0e564f527b4ada1e5cccc8e899e0d3f91b5a8" Oct 30 21:45:16.009637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:16.009545 4926 scope.go:122] "RemoveContainer" containerID="42b7a51cafd8c81856679c474bd70cea9e5073888226dc5c73d0b85639212621" Oct 30 21:45:16.009637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:16.009627 4926 scope.go:122] "RemoveContainer" containerID="e29e96aca2e4c77f84a463b63f0856543325584f26014a52467ca1e041ceac9b" Oct 30 21:45:16.010124 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:16.009755 4926 scope.go:122] "RemoveContainer" containerID="0c6d61d16b7593cd1fa651ce2442f362642b69ea6cda338470b7897a852a6e91" Oct 30 21:45:16.010124 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:16.009842 4926 scope.go:122] "RemoveContainer" containerID="1eb6abaafcbc093b287fe49f396d1612e62007d294774d4ec338f6fc0134f186" Oct 30 21:45:16.309112 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:16.309079 4926 generic.go:356] "Generic (PLEG): container finished" podID="596e496a-35f5-483f-bbde-62e3dc028766" containerID="e3b9f5168370b5efbc8738e6d82461b183898cfb24e2ce57fa56ad7befab313c" exitCode=60 Oct 30 21:45:16.309276 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:16.309160 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds" event={"ID":"596e496a-35f5-483f-bbde-62e3dc028766","Type":"ContainerDied","Data":"e3b9f5168370b5efbc8738e6d82461b183898cfb24e2ce57fa56ad7befab313c"} Oct 30 21:45:16.309276 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:16.309224 4926 scope.go:122] "RemoveContainer" containerID="3b737135da42794cbea19f17d00efd10b4110c6a9a6ee87e66f26e95621540c6" Oct 30 21:45:16.309687 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:16.309667 4926 scope.go:122] "RemoveContainer" containerID="e3b9f5168370b5efbc8738e6d82461b183898cfb24e2ce57fa56ad7befab313c" Oct 30 21:45:16.310113 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:16.310079 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds_odh-ai-gateway-infra(596e496a-35f5-483f-bbde-62e3dc028766)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds" podUID="596e496a-35f5-483f-bbde-62e3dc028766" Oct 30 21:45:17.009450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.009402 4926 scope.go:122] "RemoveContainer" containerID="f735de216a256ec53002002645601c5a4ece70d8e4156703fd842a483a830020" Oct 30 21:45:17.009644 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.009582 4926 scope.go:122] "RemoveContainer" containerID="a20ef0ccb0dfbdeed8eabe7d824b1cb32d27953c03ea4e289aa2fe00cb2aec15" Oct 30 21:45:17.322825 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.322786 4926 generic.go:356] "Generic (PLEG): container finished" podID="b0bb7714-22f3-4fe2-887c-71d638b4d61d" containerID="b28932c12c487ebf81057ef1633e2bc7499c1240c62af2b87b4dad383aabf026" exitCode=60 Oct 30 21:45:17.322984 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.322854 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n" event={"ID":"b0bb7714-22f3-4fe2-887c-71d638b4d61d","Type":"ContainerDied","Data":"b28932c12c487ebf81057ef1633e2bc7499c1240c62af2b87b4dad383aabf026"} Oct 30 21:45:17.322984 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.322918 4926 scope.go:122] "RemoveContainer" containerID="e29e96aca2e4c77f84a463b63f0856543325584f26014a52467ca1e041ceac9b" Oct 30 21:45:17.323471 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.323448 4926 scope.go:122] "RemoveContainer" containerID="b28932c12c487ebf81057ef1633e2bc7499c1240c62af2b87b4dad383aabf026" Oct 30 21:45:17.323890 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:17.323862 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n_odh-ai-gateway-infra(b0bb7714-22f3-4fe2-887c-71d638b4d61d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n" podUID="b0bb7714-22f3-4fe2-887c-71d638b4d61d" Oct 30 21:45:17.325467 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.325443 4926 generic.go:356] "Generic (PLEG): container finished" podID="2a03d85d-cd11-4d9a-ae67-29ced159f159" containerID="2d65fc8d7f226330a7eea262a5a72419d83e26fa464a1cb8181e20c851f73113" exitCode=60 Oct 30 21:45:17.325570 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.325524 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k" event={"ID":"2a03d85d-cd11-4d9a-ae67-29ced159f159","Type":"ContainerDied","Data":"2d65fc8d7f226330a7eea262a5a72419d83e26fa464a1cb8181e20c851f73113"} Oct 30 21:45:17.325939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.325916 4926 scope.go:122] "RemoveContainer" containerID="2d65fc8d7f226330a7eea262a5a72419d83e26fa464a1cb8181e20c851f73113" Oct 30 21:45:17.326300 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:17.326281 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k_odh-ai-gateway-infra(2a03d85d-cd11-4d9a-ae67-29ced159f159)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k" podUID="2a03d85d-cd11-4d9a-ae67-29ced159f159" Oct 30 21:45:17.327917 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.327891 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p" event={"ID":"c51dea80-dea4-4195-8c0a-8f28ea265669","Type":"ContainerStarted","Data":"66508593b00966fe1f02a6321574c4b994b8724c0df6d2fa57a63dbfa27d04ae"} Oct 30 21:45:17.330210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.330176 4926 generic.go:356] "Generic (PLEG): container finished" podID="c61bf7ef-856d-4d2e-b4ca-402028fabd8f" containerID="e479e22374c84304f6a7ab76c7d217e307cb48277f3dbfe50236eb2b493c4234" exitCode=60 Oct 30 21:45:17.330310 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.330220 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t" event={"ID":"c61bf7ef-856d-4d2e-b4ca-402028fabd8f","Type":"ContainerDied","Data":"e479e22374c84304f6a7ab76c7d217e307cb48277f3dbfe50236eb2b493c4234"} Oct 30 21:45:17.330754 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.330728 4926 scope.go:122] "RemoveContainer" containerID="e479e22374c84304f6a7ab76c7d217e307cb48277f3dbfe50236eb2b493c4234" Oct 30 21:45:17.331086 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:17.331053 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t_odh-ai-gateway-infra(c61bf7ef-856d-4d2e-b4ca-402028fabd8f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t" podUID="c61bf7ef-856d-4d2e-b4ca-402028fabd8f" Oct 30 21:45:17.332400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.332377 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm" event={"ID":"205a1101-2a99-4800-84d8-56a39d3690b6","Type":"ContainerStarted","Data":"19dee6f050306eb9e868805aa3da71a98c692cbb6030a5c4a768b1fc1d8f2e38"} Oct 30 21:45:17.364289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.364258 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds"] Oct 30 21:45:17.480360 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.480330 4926 scope.go:122] "RemoveContainer" containerID="0c6d61d16b7593cd1fa651ce2442f362642b69ea6cda338470b7897a852a6e91" Oct 30 21:45:17.593328 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:17.593285 4926 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51dea80_dea4_4195_8c0a_8f28ea265669.slice/crio-conmon-66508593b00966fe1f02a6321574c4b994b8724c0df6d2fa57a63dbfa27d04ae.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51dea80_dea4_4195_8c0a_8f28ea265669.slice/crio-66508593b00966fe1f02a6321574c4b994b8724c0df6d2fa57a63dbfa27d04ae.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:45:17.602717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.602691 4926 scope.go:122] "RemoveContainer" containerID="1eb6abaafcbc093b287fe49f396d1612e62007d294774d4ec338f6fc0134f186" Oct 30 21:45:17.606698 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.606672 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds" Oct 30 21:45:17.718168 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.718130 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/596e496a-35f5-483f-bbde-62e3dc028766-kube-api-access-rbmzd\" (UniqueName: \"kubernetes.io/projected/596e496a-35f5-483f-bbde-62e3dc028766-kube-api-access-rbmzd\") pod \"596e496a-35f5-483f-bbde-62e3dc028766\" (UID: \"596e496a-35f5-483f-bbde-62e3dc028766\") " Oct 30 21:45:17.718401 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.718372 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/596e496a-35f5-483f-bbde-62e3dc028766-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/596e496a-35f5-483f-bbde-62e3dc028766-maas-api-service-ca\") pod \"596e496a-35f5-483f-bbde-62e3dc028766\" (UID: \"596e496a-35f5-483f-bbde-62e3dc028766\") " Oct 30 21:45:17.718746 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.718721 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/596e496a-35f5-483f-bbde-62e3dc028766-maas-api-service-ca" pod "596e496a-35f5-483f-bbde-62e3dc028766" (UID: "596e496a-35f5-483f-bbde-62e3dc028766"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:17.718916 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.718900 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/596e496a-35f5-483f-bbde-62e3dc028766-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:17.720452 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.720430 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/596e496a-35f5-483f-bbde-62e3dc028766-kube-api-access-rbmzd" pod "596e496a-35f5-483f-bbde-62e3dc028766" (UID: "596e496a-35f5-483f-bbde-62e3dc028766"). InnerVolumeSpecName "kube-api-access-rbmzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:17.820431 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:17.820347 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbmzd\" (UniqueName: \"kubernetes.io/projected/596e496a-35f5-483f-bbde-62e3dc028766-kube-api-access-rbmzd\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:18.009319 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.009268 4926 scope.go:122] "RemoveContainer" containerID="19e35d824f0b11b5caf3db896ca0c2a72917d786c6b891a3a19befa71a1c705f" Oct 30 21:45:18.009529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.009403 4926 scope.go:122] "RemoveContainer" containerID="641a45476084c85176ebe060d1d723510f7c8ffb4506520b23abaffc3e4f4177" Oct 30 21:45:18.010084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.009932 4926 scope.go:122] "RemoveContainer" containerID="2173966983d96371216428a33372bd3de88436c396279849db4548ab16b5c80a" Oct 30 21:45:18.211862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.211827 4926 scope.go:122] "RemoveContainer" containerID="525c404701b1ecd1b51542cfdbf2ddb2b4ad77df5d1052e1cd6d0f8bed27277f" Oct 30 21:45:18.224989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.224963 4926 scope.go:122] "RemoveContainer" containerID="bf4cdd9ed92c64d220a702d0e7201624eb6f07b437a903c6b38c8401fe8b4727" Oct 30 21:45:18.349246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.349154 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl"] Oct 30 21:45:18.349423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.349249 4926 generic.go:356] "Generic (PLEG): container finished" podID="205a1101-2a99-4800-84d8-56a39d3690b6" containerID="19dee6f050306eb9e868805aa3da71a98c692cbb6030a5c4a768b1fc1d8f2e38" exitCode=60 Oct 30 21:45:18.350087 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.350058 4926 cpu_manager.go:414] "containerMap: removing container" podUID="596e496a-35f5-483f-bbde-62e3dc028766" containerName="cleanup" Oct 30 21:45:18.350087 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.350081 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596e496a-35f5-483f-bbde-62e3dc028766" containerName="cleanup" Oct 30 21:45:18.350273 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.350119 4926 cpu_manager.go:414] "containerMap: removing container" podUID="596e496a-35f5-483f-bbde-62e3dc028766" containerName="cleanup" Oct 30 21:45:18.350273 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.350127 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596e496a-35f5-483f-bbde-62e3dc028766" containerName="cleanup" Oct 30 21:45:18.350273 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.350148 4926 cpu_manager.go:414] "containerMap: removing container" podUID="596e496a-35f5-483f-bbde-62e3dc028766" containerName="cleanup" Oct 30 21:45:18.350273 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.350156 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="596e496a-35f5-483f-bbde-62e3dc028766" containerName="cleanup" Oct 30 21:45:18.350470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.350347 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="596e496a-35f5-483f-bbde-62e3dc028766" containerName="cleanup" Oct 30 21:45:18.350470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.350364 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="596e496a-35f5-483f-bbde-62e3dc028766" containerName="cleanup" Oct 30 21:45:18.350470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.350377 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="596e496a-35f5-483f-bbde-62e3dc028766" containerName="cleanup" Oct 30 21:45:18.351243 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.351219 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds" Oct 30 21:45:18.357087 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.356422 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm" event={"ID":"205a1101-2a99-4800-84d8-56a39d3690b6","Type":"ContainerDied","Data":"19dee6f050306eb9e868805aa3da71a98c692cbb6030a5c4a768b1fc1d8f2e38"} Oct 30 21:45:18.357087 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.356472 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds" event={"ID":"596e496a-35f5-483f-bbde-62e3dc028766","Type":"ContainerDied","Data":"a3f78e900443de35f32ee2a152ec52962f4a34212cbf2ea677b06b4dc2057d54"} Oct 30 21:45:18.357087 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.356513 4926 scope.go:122] "RemoveContainer" containerID="42b7a51cafd8c81856679c474bd70cea9e5073888226dc5c73d0b85639212621" Oct 30 21:45:18.357087 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.356629 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:18.385434 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.359822 4926 scope.go:122] "RemoveContainer" containerID="19dee6f050306eb9e868805aa3da71a98c692cbb6030a5c4a768b1fc1d8f2e38" Oct 30 21:45:18.385434 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:18.360204 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm_odh-ai-gateway-infra(205a1101-2a99-4800-84d8-56a39d3690b6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm" podUID="205a1101-2a99-4800-84d8-56a39d3690b6" Oct 30 21:45:18.385434 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.370372 4926 generic.go:356] "Generic (PLEG): container finished" podID="c51dea80-dea4-4195-8c0a-8f28ea265669" containerID="66508593b00966fe1f02a6321574c4b994b8724c0df6d2fa57a63dbfa27d04ae" exitCode=60 Oct 30 21:45:18.385434 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.370453 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p" event={"ID":"c51dea80-dea4-4195-8c0a-8f28ea265669","Type":"ContainerDied","Data":"66508593b00966fe1f02a6321574c4b994b8724c0df6d2fa57a63dbfa27d04ae"} Oct 30 21:45:18.385434 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.379662 4926 scope.go:122] "RemoveContainer" containerID="66508593b00966fe1f02a6321574c4b994b8724c0df6d2fa57a63dbfa27d04ae" Oct 30 21:45:18.385434 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:18.380093 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p_odh-ai-gateway-infra(c51dea80-dea4-4195-8c0a-8f28ea265669)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p" podUID="c51dea80-dea4-4195-8c0a-8f28ea265669" Oct 30 21:45:18.385434 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.381613 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-908469\"/\"default-dockercfg-mvdmg\"" Oct 30 21:45:18.385434 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.382319 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-908469\"/\"kube-root-ca.crt\"" Oct 30 21:45:18.385434 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.382408 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-908469\"/\"openshift-service-ca.crt\"" Oct 30 21:45:18.387648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.387623 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl"] Oct 30 21:45:18.403386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.403290 4926 scope.go:122] "RemoveContainer" containerID="e3b9f5168370b5efbc8738e6d82461b183898cfb24e2ce57fa56ad7befab313c" Oct 30 21:45:18.404881 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.404858 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n"] Oct 30 21:45:18.407625 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.407399 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn" event={"ID":"19dafab6-6daa-4e31-bc89-3ebd04adffbb","Type":"ContainerStarted","Data":"fbef41a795a4bf5f846dee6dc93fe7b73f8611bf0a348c46b5d01d8f07add044"} Oct 30 21:45:18.407909 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.407880 4926 scope.go:122] "RemoveContainer" containerID="fbef41a795a4bf5f846dee6dc93fe7b73f8611bf0a348c46b5d01d8f07add044" Oct 30 21:45:18.408523 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:18.408495 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn_odh-ai-gateway-infra(19dafab6-6daa-4e31-bc89-3ebd04adffbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn" podUID="19dafab6-6daa-4e31-bc89-3ebd04adffbb" Oct 30 21:45:18.424187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.424153 4926 generic.go:356] "Generic (PLEG): container finished" podID="e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3" containerID="8128c007380eb925fb93fa3f15a5e4f1cad6faf87f92710d5991f09f8fba701d" exitCode=60 Oct 30 21:45:18.424407 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.424196 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz" event={"ID":"e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3","Type":"ContainerDied","Data":"8128c007380eb925fb93fa3f15a5e4f1cad6faf87f92710d5991f09f8fba701d"} Oct 30 21:45:18.424707 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.424664 4926 scope.go:122] "RemoveContainer" containerID="8128c007380eb925fb93fa3f15a5e4f1cad6faf87f92710d5991f09f8fba701d" Oct 30 21:45:18.425088 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:18.425025 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz_odh-ai-gateway-infra(e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz" podUID="e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3" Oct 30 21:45:18.429429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.429400 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9e51cd-e4a7-4944-a1da-23a596732dbb-tls-certs\") pod \"sub-test-model-908469-kserve-5b74b5c669-zwhbl\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:18.432443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.429686 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-home\") pod \"sub-test-model-908469-kserve-5b74b5c669-zwhbl\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:18.432443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.429749 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-dshm\") pod \"sub-test-model-908469-kserve-5b74b5c669-zwhbl\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:18.432443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.429786 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-tmp-dir\") pod \"sub-test-model-908469-kserve-5b74b5c669-zwhbl\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:18.432443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.429842 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-model-cache\") pod \"sub-test-model-908469-kserve-5b74b5c669-zwhbl\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:18.432443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.429901 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnjz8\" (UniqueName: \"kubernetes.io/projected/7d9e51cd-e4a7-4944-a1da-23a596732dbb-kube-api-access-wnjz8\") pod \"sub-test-model-908469-kserve-5b74b5c669-zwhbl\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:18.444833 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.444798 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t"] Oct 30 21:45:18.460374 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.460342 4926 scope.go:122] "RemoveContainer" containerID="78f35822e24ae935edbfdafa47a0e564f527b4ada1e5cccc8e899e0d3f91b5a8" Oct 30 21:45:18.489460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.481888 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-908469\"/\"sub-test-model-908469-kserve-self-signed-certs\"" Oct 30 21:45:18.521069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.521011 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p"] Oct 30 21:45:18.525076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.524244 4926 scope.go:122] "RemoveContainer" containerID="a20ef0ccb0dfbdeed8eabe7d824b1cb32d27953c03ea4e289aa2fe00cb2aec15" Oct 30 21:45:18.553500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.539926 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-dshm\") pod \"sub-test-model-908469-kserve-5b74b5c669-zwhbl\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:18.553500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.539966 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-tmp-dir\") pod \"sub-test-model-908469-kserve-5b74b5c669-zwhbl\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:18.553500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.540738 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-tmp-dir\") pod \"sub-test-model-908469-kserve-5b74b5c669-zwhbl\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:18.553500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.541077 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-model-cache\") pod \"sub-test-model-908469-kserve-5b74b5c669-zwhbl\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:18.553500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.541181 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnjz8\" (UniqueName: \"kubernetes.io/projected/7d9e51cd-e4a7-4944-a1da-23a596732dbb-kube-api-access-wnjz8\") pod \"sub-test-model-908469-kserve-5b74b5c669-zwhbl\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:18.553500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.541234 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9e51cd-e4a7-4944-a1da-23a596732dbb-tls-certs\") pod \"sub-test-model-908469-kserve-5b74b5c669-zwhbl\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:18.553500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.541484 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-home\") pod \"sub-test-model-908469-kserve-5b74b5c669-zwhbl\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:18.553500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.542457 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-model-cache\") pod \"sub-test-model-908469-kserve-5b74b5c669-zwhbl\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:18.553500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.543574 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-home\") pod \"sub-test-model-908469-kserve-5b74b5c669-zwhbl\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:18.553500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.546184 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-dshm\") pod \"sub-test-model-908469-kserve-5b74b5c669-zwhbl\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:18.553500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.547683 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9e51cd-e4a7-4944-a1da-23a596732dbb-tls-certs\") pod \"sub-test-model-908469-kserve-5b74b5c669-zwhbl\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:18.576433 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.574161 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnjz8\" (UniqueName: \"kubernetes.io/projected/7d9e51cd-e4a7-4944-a1da-23a596732dbb-kube-api-access-wnjz8\") pod \"sub-test-model-908469-kserve-5b74b5c669-zwhbl\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:18.849057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.790602 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:18.851002 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.850932 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k"] Oct 30 21:45:18.912940 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.907103 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm"] Oct 30 21:45:18.913417 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.913386 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n" Oct 30 21:45:18.966491 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.966451 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t" Oct 30 21:45:18.967001 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.966964 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds"] Oct 30 21:45:18.978141 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:18.978077 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-79477e-29889945-mcbds"] Oct 30 21:45:19.018386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.018237 4926 scope.go:122] "RemoveContainer" containerID="76ea305952dc23ebc2bb8988d1bf45e9b520839b3056068112e077e266273ec5" Oct 30 21:45:19.019980 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.019958 4926 scope.go:122] "RemoveContainer" containerID="918ad2c9e7b0c1fa9fe04748f27d0c432841e6d5812fa9e39ceb8531a39de71c" Oct 30 21:45:19.022655 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.021943 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0bb7714-22f3-4fe2-887c-71d638b4d61d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bb7714-22f3-4fe2-887c-71d638b4d61d-maas-api-service-ca\") pod \"b0bb7714-22f3-4fe2-887c-71d638b4d61d\" (UID: \"b0bb7714-22f3-4fe2-887c-71d638b4d61d\") " Oct 30 21:45:19.022853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.022834 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c61bf7ef-856d-4d2e-b4ca-402028fabd8f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61bf7ef-856d-4d2e-b4ca-402028fabd8f-maas-api-service-ca\") pod \"c61bf7ef-856d-4d2e-b4ca-402028fabd8f\" (UID: \"c61bf7ef-856d-4d2e-b4ca-402028fabd8f\") " Oct 30 21:45:19.023050 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.023019 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bb7714-22f3-4fe2-887c-71d638b4d61d-kube-api-access-cfm2c\" (UniqueName: \"kubernetes.io/projected/b0bb7714-22f3-4fe2-887c-71d638b4d61d-kube-api-access-cfm2c\") pod \"b0bb7714-22f3-4fe2-887c-71d638b4d61d\" (UID: \"b0bb7714-22f3-4fe2-887c-71d638b4d61d\") " Oct 30 21:45:19.023394 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.023374 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61bf7ef-856d-4d2e-b4ca-402028fabd8f-kube-api-access-brfrf\" (UniqueName: \"kubernetes.io/projected/c61bf7ef-856d-4d2e-b4ca-402028fabd8f-kube-api-access-brfrf\") pod \"c61bf7ef-856d-4d2e-b4ca-402028fabd8f\" (UID: \"c61bf7ef-856d-4d2e-b4ca-402028fabd8f\") " Oct 30 21:45:19.024243 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.024219 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bb7714-22f3-4fe2-887c-71d638b4d61d-maas-api-service-ca" pod "b0bb7714-22f3-4fe2-887c-71d638b4d61d" (UID: "b0bb7714-22f3-4fe2-887c-71d638b4d61d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:19.038354 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.033010 4926 scope.go:122] "RemoveContainer" containerID="6f4b3e32e7c710b63e4e97b756538732f295281196d6caeeb7eaf0456549f0ed" Oct 30 21:45:19.038354 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.033993 4926 scope.go:122] "RemoveContainer" containerID="0e112c22d0d6cd6e8ab8cff8fe0f8d4562c9cfbbf11bb0590612fa394391dcc7" Oct 30 21:45:19.042145 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.040881 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61bf7ef-856d-4d2e-b4ca-402028fabd8f-maas-api-service-ca" pod "c61bf7ef-856d-4d2e-b4ca-402028fabd8f" (UID: "c61bf7ef-856d-4d2e-b4ca-402028fabd8f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:19.044336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.044284 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="596e496a-35f5-483f-bbde-62e3dc028766" path="/var/lib/kubelet/pods/596e496a-35f5-483f-bbde-62e3dc028766/volumes" Oct 30 21:45:19.063809 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.063739 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61bf7ef-856d-4d2e-b4ca-402028fabd8f-kube-api-access-brfrf" pod "c61bf7ef-856d-4d2e-b4ca-402028fabd8f" (UID: "c61bf7ef-856d-4d2e-b4ca-402028fabd8f"). InnerVolumeSpecName "kube-api-access-brfrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:19.071433 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.071379 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bb7714-22f3-4fe2-887c-71d638b4d61d-kube-api-access-cfm2c" pod "b0bb7714-22f3-4fe2-887c-71d638b4d61d" (UID: "b0bb7714-22f3-4fe2-887c-71d638b4d61d"). InnerVolumeSpecName "kube-api-access-cfm2c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:19.130105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.127284 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0bb7714-22f3-4fe2-887c-71d638b4d61d-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:19.130105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.127320 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c61bf7ef-856d-4d2e-b4ca-402028fabd8f-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:19.130105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.127335 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfm2c\" (UniqueName: \"kubernetes.io/projected/b0bb7714-22f3-4fe2-887c-71d638b4d61d-kube-api-access-cfm2c\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:19.130105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.127349 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brfrf\" (UniqueName: \"kubernetes.io/projected/c61bf7ef-856d-4d2e-b4ca-402028fabd8f-kube-api-access-brfrf\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:19.232515 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.230754 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl"] Oct 30 21:45:19.274503 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:45:19.274391 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d9e51cd_e4a7_4944_a1da_23a596732dbb.slice/crio-0cdc529ec746db23c2b2a1eb1882dcaeb9e1d8991f72c648007746edeab10553 WatchSource:0}: Error finding container 0cdc529ec746db23c2b2a1eb1882dcaeb9e1d8991f72c648007746edeab10553: Status 404 returned error can't find the container with id 0cdc529ec746db23c2b2a1eb1882dcaeb9e1d8991f72c648007746edeab10553 Oct 30 21:45:19.325697 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.325661 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k" Oct 30 21:45:19.432972 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.432909 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a03d85d-cd11-4d9a-ae67-29ced159f159-kube-api-access-z5g6m\" (UniqueName: \"kubernetes.io/projected/2a03d85d-cd11-4d9a-ae67-29ced159f159-kube-api-access-z5g6m\") pod \"2a03d85d-cd11-4d9a-ae67-29ced159f159\" (UID: \"2a03d85d-cd11-4d9a-ae67-29ced159f159\") " Oct 30 21:45:19.433647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.433604 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a03d85d-cd11-4d9a-ae67-29ced159f159-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a03d85d-cd11-4d9a-ae67-29ced159f159-maas-api-service-ca\") pod \"2a03d85d-cd11-4d9a-ae67-29ced159f159\" (UID: \"2a03d85d-cd11-4d9a-ae67-29ced159f159\") " Oct 30 21:45:19.434394 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.434365 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a03d85d-cd11-4d9a-ae67-29ced159f159-maas-api-service-ca" pod "2a03d85d-cd11-4d9a-ae67-29ced159f159" (UID: "2a03d85d-cd11-4d9a-ae67-29ced159f159"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:19.439904 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.439870 4926 generic.go:356] "Generic (PLEG): container finished" podID="f00eb47b-3c81-4420-a948-45dae9800548" containerID="30a3c16ee5295ab014e60d3679af070ac7bde249291a03137a3e7de90d800288" exitCode=60 Oct 30 21:45:19.440078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.439897 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t" event={"ID":"f00eb47b-3c81-4420-a948-45dae9800548","Type":"ContainerDied","Data":"30a3c16ee5295ab014e60d3679af070ac7bde249291a03137a3e7de90d800288"} Oct 30 21:45:19.440078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.439987 4926 scope.go:122] "RemoveContainer" containerID="2173966983d96371216428a33372bd3de88436c396279849db4548ab16b5c80a" Oct 30 21:45:19.440494 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.440464 4926 scope.go:122] "RemoveContainer" containerID="30a3c16ee5295ab014e60d3679af070ac7bde249291a03137a3e7de90d800288" Oct 30 21:45:19.440852 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:19.440831 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t_odh-ai-gateway-infra(f00eb47b-3c81-4420-a948-45dae9800548)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t" podUID="f00eb47b-3c81-4420-a948-45dae9800548" Oct 30 21:45:19.442219 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.442160 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" event={"ID":"7d9e51cd-e4a7-4944-a1da-23a596732dbb","Type":"ContainerStarted","Data":"ab7af3be6c85ed08a618bceeb207a8d5ab5302a47d05f5cb18c9953ecd42881b"} Oct 30 21:45:19.442378 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.442356 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" event={"ID":"7d9e51cd-e4a7-4944-a1da-23a596732dbb","Type":"ContainerStarted","Data":"0cdc529ec746db23c2b2a1eb1882dcaeb9e1d8991f72c648007746edeab10553"} Oct 30 21:45:19.443984 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.443965 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t" Oct 30 21:45:19.444574 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.444520 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t" event={"ID":"c61bf7ef-856d-4d2e-b4ca-402028fabd8f","Type":"ContainerDied","Data":"a2f2b8f0ec6c13b1eb47d793c145dccb38735c688b1082262835ac443dc8c73e"} Oct 30 21:45:19.452653 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.452623 4926 generic.go:356] "Generic (PLEG): container finished" podID="30def38d-8a67-4281-9031-c6f7f7c96bc5" containerID="8e3d8f79a606bc524d6583edaebaddc69e010aa68bf8e97b04040f6b176b092c" exitCode=60 Oct 30 21:45:19.452784 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.452694 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4" event={"ID":"30def38d-8a67-4281-9031-c6f7f7c96bc5","Type":"ContainerDied","Data":"8e3d8f79a606bc524d6583edaebaddc69e010aa68bf8e97b04040f6b176b092c"} Oct 30 21:45:19.455900 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.455875 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n" event={"ID":"b0bb7714-22f3-4fe2-887c-71d638b4d61d","Type":"ContainerDied","Data":"c3fe9d7b605418a5029849a541abc1cb4708b075bd1dd1e68e6ff5534f34cfa4"} Oct 30 21:45:19.455989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.455968 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n" Oct 30 21:45:19.456173 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.456149 4926 scope.go:122] "RemoveContainer" containerID="8e3d8f79a606bc524d6583edaebaddc69e010aa68bf8e97b04040f6b176b092c" Oct 30 21:45:19.456550 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:19.456525 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4_odh-ai-gateway-infra(30def38d-8a67-4281-9031-c6f7f7c96bc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4" podUID="30def38d-8a67-4281-9031-c6f7f7c96bc5" Oct 30 21:45:19.458690 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.458627 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k" event={"ID":"2a03d85d-cd11-4d9a-ae67-29ced159f159","Type":"ContainerDied","Data":"34dd91abe0deb24548bf5f937062ba40f8cb35a98e9ff17754c0e67aa4fae8c0"} Oct 30 21:45:19.459281 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.458834 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k" Oct 30 21:45:19.459765 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.459680 4926 scope.go:122] "RemoveContainer" containerID="e479e22374c84304f6a7ab76c7d217e307cb48277f3dbfe50236eb2b493c4234" Oct 30 21:45:19.462586 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.462555 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj" event={"ID":"657d2744-b23d-4dbb-a75d-23fea5f9a286","Type":"ContainerStarted","Data":"2f40956b289a75d382627b11846ce4f0771913920a04630e467819958797b4a5"} Oct 30 21:45:19.472959 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.472922 4926 generic.go:356] "Generic (PLEG): container finished" podID="19dafab6-6daa-4e31-bc89-3ebd04adffbb" containerID="fbef41a795a4bf5f846dee6dc93fe7b73f8611bf0a348c46b5d01d8f07add044" exitCode=60 Oct 30 21:45:19.473154 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.472988 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn" event={"ID":"19dafab6-6daa-4e31-bc89-3ebd04adffbb","Type":"ContainerDied","Data":"fbef41a795a4bf5f846dee6dc93fe7b73f8611bf0a348c46b5d01d8f07add044"} Oct 30 21:45:19.473786 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.473772 4926 scope.go:122] "RemoveContainer" containerID="fbef41a795a4bf5f846dee6dc93fe7b73f8611bf0a348c46b5d01d8f07add044" Oct 30 21:45:19.474467 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:19.474446 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn_odh-ai-gateway-infra(19dafab6-6daa-4e31-bc89-3ebd04adffbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn" podUID="19dafab6-6daa-4e31-bc89-3ebd04adffbb" Oct 30 21:45:19.478473 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.478434 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a03d85d-cd11-4d9a-ae67-29ced159f159-kube-api-access-z5g6m" pod "2a03d85d-cd11-4d9a-ae67-29ced159f159" (UID: "2a03d85d-cd11-4d9a-ae67-29ced159f159"). InnerVolumeSpecName "kube-api-access-z5g6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:19.480088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.478906 4926 scope.go:122] "RemoveContainer" containerID="641a45476084c85176ebe060d1d723510f7c8ffb4506520b23abaffc3e4f4177" Oct 30 21:45:19.536485 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.536451 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a03d85d-cd11-4d9a-ae67-29ced159f159-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:19.536485 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.536486 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5g6m\" (UniqueName: \"kubernetes.io/projected/2a03d85d-cd11-4d9a-ae67-29ced159f159-kube-api-access-z5g6m\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:19.572045 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.570362 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" podStartSLOduration=1.570338615 podStartE2EDuration="1.570338615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:19.56387764 +0000 UTC m=+2706.684951387" watchObservedRunningTime="2026-10-30 21:45:19.570338615 +0000 UTC m=+2706.691412362" Oct 30 21:45:19.610649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.610615 4926 scope.go:122] "RemoveContainer" containerID="b28932c12c487ebf81057ef1633e2bc7499c1240c62af2b87b4dad383aabf026" Oct 30 21:45:19.626678 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.626421 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t"] Oct 30 21:45:19.628614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.628504 4926 scope.go:122] "RemoveContainer" containerID="2d65fc8d7f226330a7eea262a5a72419d83e26fa464a1cb8181e20c851f73113" Oct 30 21:45:19.638022 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.637604 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ed088d-29889945-p6j2t"] Oct 30 21:45:19.661871 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.661767 4926 scope.go:122] "RemoveContainer" containerID="f735de216a256ec53002002645601c5a4ece70d8e4156703fd842a483a830020" Oct 30 21:45:19.681165 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.681134 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p" Oct 30 21:45:19.720730 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.720612 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n"] Oct 30 21:45:19.726050 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.725996 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-a4782a-29889945-s2t6n"] Oct 30 21:45:19.732208 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.732175 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz"] Oct 30 21:45:19.739423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.739390 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c51dea80-dea4-4195-8c0a-8f28ea265669-kube-api-access-7lpqv\" (UniqueName: \"kubernetes.io/projected/c51dea80-dea4-4195-8c0a-8f28ea265669-kube-api-access-7lpqv\") pod \"c51dea80-dea4-4195-8c0a-8f28ea265669\" (UID: \"c51dea80-dea4-4195-8c0a-8f28ea265669\") " Oct 30 21:45:19.739575 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.739553 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c51dea80-dea4-4195-8c0a-8f28ea265669-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c51dea80-dea4-4195-8c0a-8f28ea265669-maas-api-service-ca\") pod \"c51dea80-dea4-4195-8c0a-8f28ea265669\" (UID: \"c51dea80-dea4-4195-8c0a-8f28ea265669\") " Oct 30 21:45:19.739973 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.739945 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51dea80-dea4-4195-8c0a-8f28ea265669-maas-api-service-ca" pod "c51dea80-dea4-4195-8c0a-8f28ea265669" (UID: "c51dea80-dea4-4195-8c0a-8f28ea265669"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:19.740266 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.740246 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c51dea80-dea4-4195-8c0a-8f28ea265669-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:19.779226 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.778535 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51dea80-dea4-4195-8c0a-8f28ea265669-kube-api-access-7lpqv" pod "c51dea80-dea4-4195-8c0a-8f28ea265669" (UID: "c51dea80-dea4-4195-8c0a-8f28ea265669"). InnerVolumeSpecName "kube-api-access-7lpqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:19.805659 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.796909 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:19.842236 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.842191 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lpqv\" (UniqueName: \"kubernetes.io/projected/c51dea80-dea4-4195-8c0a-8f28ea265669-kube-api-access-7lpqv\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:19.910848 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.910793 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm" Oct 30 21:45:19.920246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.920183 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj" podStartSLOduration=19.920161339 podStartE2EDuration="19.920161339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:19.918961677 +0000 UTC m=+2707.040035422" watchObservedRunningTime="2026-10-30 21:45:19.920161339 +0000 UTC m=+2707.041235087" Oct 30 21:45:19.951115 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.951079 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz" Oct 30 21:45:19.967935 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:19.967899 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn"] Oct 30 21:45:20.044248 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.044157 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3-kube-api-access-75tlf\" (UniqueName: \"kubernetes.io/projected/e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3-kube-api-access-75tlf\") pod \"e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3\" (UID: \"e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3\") " Oct 30 21:45:20.044794 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.044282 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3-maas-api-service-ca\") pod \"e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3\" (UID: \"e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3\") " Oct 30 21:45:20.044794 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.044359 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/205a1101-2a99-4800-84d8-56a39d3690b6-kube-api-access-jnbxr\" (UniqueName: \"kubernetes.io/projected/205a1101-2a99-4800-84d8-56a39d3690b6-kube-api-access-jnbxr\") pod \"205a1101-2a99-4800-84d8-56a39d3690b6\" (UID: \"205a1101-2a99-4800-84d8-56a39d3690b6\") " Oct 30 21:45:20.044794 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.044575 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/205a1101-2a99-4800-84d8-56a39d3690b6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/205a1101-2a99-4800-84d8-56a39d3690b6-maas-api-service-ca\") pod \"205a1101-2a99-4800-84d8-56a39d3690b6\" (UID: \"205a1101-2a99-4800-84d8-56a39d3690b6\") " Oct 30 21:45:20.044794 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.044658 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3-maas-api-service-ca" pod "e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3" (UID: "e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:20.045028 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.044961 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/205a1101-2a99-4800-84d8-56a39d3690b6-maas-api-service-ca" pod "205a1101-2a99-4800-84d8-56a39d3690b6" (UID: "205a1101-2a99-4800-84d8-56a39d3690b6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:20.045223 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.045204 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/205a1101-2a99-4800-84d8-56a39d3690b6-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:20.045298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.045228 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:20.047082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.047024 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3-kube-api-access-75tlf" pod "e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3" (UID: "e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3"). InnerVolumeSpecName "kube-api-access-75tlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:20.047158 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.047095 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/205a1101-2a99-4800-84d8-56a39d3690b6-kube-api-access-jnbxr" pod "205a1101-2a99-4800-84d8-56a39d3690b6" (UID: "205a1101-2a99-4800-84d8-56a39d3690b6"). InnerVolumeSpecName "kube-api-access-jnbxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:20.152692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.150124 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75tlf\" (UniqueName: \"kubernetes.io/projected/e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3-kube-api-access-75tlf\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:20.152692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.150165 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnbxr\" (UniqueName: \"kubernetes.io/projected/205a1101-2a99-4800-84d8-56a39d3690b6-kube-api-access-jnbxr\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:20.189432 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.189394 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k"] Oct 30 21:45:20.198184 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.198137 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-c46397-29889945-xzb4k"] Oct 30 21:45:20.491128 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.491079 4926 generic.go:356] "Generic (PLEG): container finished" podID="657d2744-b23d-4dbb-a75d-23fea5f9a286" containerID="2f40956b289a75d382627b11846ce4f0771913920a04630e467819958797b4a5" exitCode=60 Oct 30 21:45:20.491288 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.491155 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj" event={"ID":"657d2744-b23d-4dbb-a75d-23fea5f9a286","Type":"ContainerDied","Data":"2f40956b289a75d382627b11846ce4f0771913920a04630e467819958797b4a5"} Oct 30 21:45:20.491288 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.491276 4926 scope.go:122] "RemoveContainer" containerID="19e35d824f0b11b5caf3db896ca0c2a72917d786c6b891a3a19befa71a1c705f" Oct 30 21:45:20.491799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.491778 4926 scope.go:122] "RemoveContainer" containerID="2f40956b289a75d382627b11846ce4f0771913920a04630e467819958797b4a5" Oct 30 21:45:20.492225 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:20.492199 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj_odh-ai-gateway-infra(657d2744-b23d-4dbb-a75d-23fea5f9a286)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj" podUID="657d2744-b23d-4dbb-a75d-23fea5f9a286" Oct 30 21:45:20.493123 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.493088 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p" event={"ID":"c51dea80-dea4-4195-8c0a-8f28ea265669","Type":"ContainerDied","Data":"48e98b2025034bbc918471fd1615b9aa2aab9e759a39e2affe6de3e53cbb8de5"} Oct 30 21:45:20.493216 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.493154 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p" Oct 30 21:45:20.497921 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.497892 4926 generic.go:356] "Generic (PLEG): container finished" podID="42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543" containerID="0ac7ae4197c6e7ae46dca10985e9222f3d7974895693958d23c38286ca996062" exitCode=60 Oct 30 21:45:20.498139 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.497957 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz" event={"ID":"42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543","Type":"ContainerDied","Data":"0ac7ae4197c6e7ae46dca10985e9222f3d7974895693958d23c38286ca996062"} Oct 30 21:45:20.498466 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.498448 4926 scope.go:122] "RemoveContainer" containerID="0ac7ae4197c6e7ae46dca10985e9222f3d7974895693958d23c38286ca996062" Oct 30 21:45:20.498801 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:20.498775 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz_odh-ai-gateway-infra(42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz" podUID="42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543" Oct 30 21:45:20.502099 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.502072 4926 generic.go:356] "Generic (PLEG): container finished" podID="bd055827-04e0-4946-bf79-b8e59e81fa21" containerID="1ae1846caf046ce5514f1f5b1477aa55a1c1ed18aba32f9f3c698217c88cfddf" exitCode=60 Oct 30 21:45:20.502199 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.502145 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95" event={"ID":"bd055827-04e0-4946-bf79-b8e59e81fa21","Type":"ContainerDied","Data":"1ae1846caf046ce5514f1f5b1477aa55a1c1ed18aba32f9f3c698217c88cfddf"} Oct 30 21:45:20.502591 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.502572 4926 scope.go:122] "RemoveContainer" containerID="1ae1846caf046ce5514f1f5b1477aa55a1c1ed18aba32f9f3c698217c88cfddf" Oct 30 21:45:20.503110 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:20.503088 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-5sb95_odh-ai-gateway-infra(bd055827-04e0-4946-bf79-b8e59e81fa21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95" podUID="bd055827-04e0-4946-bf79-b8e59e81fa21" Oct 30 21:45:20.507474 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.507442 4926 generic.go:356] "Generic (PLEG): container finished" podID="d0f27f31-e62c-482e-bfba-10c2250410a4" containerID="89211d446c4e209ef2336624e4b8b3e530033fd9d9556d46b44a626bccf54d03" exitCode=60 Oct 30 21:45:20.507583 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.507566 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4" event={"ID":"d0f27f31-e62c-482e-bfba-10c2250410a4","Type":"ContainerDied","Data":"89211d446c4e209ef2336624e4b8b3e530033fd9d9556d46b44a626bccf54d03"} Oct 30 21:45:20.508120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.508096 4926 scope.go:122] "RemoveContainer" containerID="89211d446c4e209ef2336624e4b8b3e530033fd9d9556d46b44a626bccf54d03" Oct 30 21:45:20.508493 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:20.508468 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4_odh-ai-gateway-infra(d0f27f31-e62c-482e-bfba-10c2250410a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4" podUID="d0f27f31-e62c-482e-bfba-10c2250410a4" Oct 30 21:45:20.511201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.511169 4926 generic.go:356] "Generic (PLEG): container finished" podID="547cf37c-665f-4c4b-ab6f-06b5422b9cc9" containerID="94bab3925c86108979fdb33accb61322b58c2e85f15f042941f4de744585b028" exitCode=60 Oct 30 21:45:20.511331 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.511312 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2" event={"ID":"547cf37c-665f-4c4b-ab6f-06b5422b9cc9","Type":"ContainerDied","Data":"94bab3925c86108979fdb33accb61322b58c2e85f15f042941f4de744585b028"} Oct 30 21:45:20.511713 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.511695 4926 scope.go:122] "RemoveContainer" containerID="94bab3925c86108979fdb33accb61322b58c2e85f15f042941f4de744585b028" Oct 30 21:45:20.512067 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:20.512009 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2_odh-ai-gateway-infra(547cf37c-665f-4c4b-ab6f-06b5422b9cc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2" podUID="547cf37c-665f-4c4b-ab6f-06b5422b9cc9" Oct 30 21:45:20.514581 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.514553 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t"] Oct 30 21:45:20.514752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.514691 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz" Oct 30 21:45:20.514983 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.514956 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz" event={"ID":"e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3","Type":"ContainerDied","Data":"876c39236f79f080758dc9671700c4481eb864da842b66764375a372de2914f6"} Oct 30 21:45:20.516771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.516750 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm" Oct 30 21:45:20.516899 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.516761 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm" event={"ID":"205a1101-2a99-4800-84d8-56a39d3690b6","Type":"ContainerDied","Data":"62dc66bcc454267c9480b53112e74494fad9e446dfe4adf9512cd03b4f4e4c9a"} Oct 30 21:45:20.578247 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.578220 4926 scope.go:122] "RemoveContainer" containerID="66508593b00966fe1f02a6321574c4b994b8724c0df6d2fa57a63dbfa27d04ae" Oct 30 21:45:20.602294 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.602267 4926 scope.go:122] "RemoveContainer" containerID="918ad2c9e7b0c1fa9fe04748f27d0c432841e6d5812fa9e39ceb8531a39de71c" Oct 30 21:45:20.628582 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.628553 4926 scope.go:122] "RemoveContainer" containerID="6f4b3e32e7c710b63e4e97b756538732f295281196d6caeeb7eaf0456549f0ed" Oct 30 21:45:20.647047 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.646986 4926 scope.go:122] "RemoveContainer" containerID="0e112c22d0d6cd6e8ab8cff8fe0f8d4562c9cfbbf11bb0590612fa394391dcc7" Oct 30 21:45:20.672573 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.672542 4926 scope.go:122] "RemoveContainer" containerID="76ea305952dc23ebc2bb8988d1bf45e9b520839b3056068112e077e266273ec5" Oct 30 21:45:20.699671 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.699643 4926 scope.go:122] "RemoveContainer" containerID="8128c007380eb925fb93fa3f15a5e4f1cad6faf87f92710d5991f09f8fba701d" Oct 30 21:45:20.721257 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.721226 4926 scope.go:122] "RemoveContainer" containerID="19dee6f050306eb9e868805aa3da71a98c692cbb6030a5c4a768b1fc1d8f2e38" Oct 30 21:45:20.736847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.736822 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn" Oct 30 21:45:20.757536 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.757507 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t" Oct 30 21:45:20.850197 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.850163 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4"] Oct 30 21:45:20.868202 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.868150 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19dafab6-6daa-4e31-bc89-3ebd04adffbb-kube-api-access-rwnsh\" (UniqueName: \"kubernetes.io/projected/19dafab6-6daa-4e31-bc89-3ebd04adffbb-kube-api-access-rwnsh\") pod \"19dafab6-6daa-4e31-bc89-3ebd04adffbb\" (UID: \"19dafab6-6daa-4e31-bc89-3ebd04adffbb\") " Oct 30 21:45:20.868375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.868253 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f00eb47b-3c81-4420-a948-45dae9800548-kube-api-access-w9rbf\" (UniqueName: \"kubernetes.io/projected/f00eb47b-3c81-4420-a948-45dae9800548-kube-api-access-w9rbf\") pod \"f00eb47b-3c81-4420-a948-45dae9800548\" (UID: \"f00eb47b-3c81-4420-a948-45dae9800548\") " Oct 30 21:45:20.868375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.868318 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19dafab6-6daa-4e31-bc89-3ebd04adffbb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19dafab6-6daa-4e31-bc89-3ebd04adffbb-maas-api-service-ca\") pod \"19dafab6-6daa-4e31-bc89-3ebd04adffbb\" (UID: \"19dafab6-6daa-4e31-bc89-3ebd04adffbb\") " Oct 30 21:45:20.868375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.868354 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f00eb47b-3c81-4420-a948-45dae9800548-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00eb47b-3c81-4420-a948-45dae9800548-maas-api-service-ca\") pod \"f00eb47b-3c81-4420-a948-45dae9800548\" (UID: \"f00eb47b-3c81-4420-a948-45dae9800548\") " Oct 30 21:45:20.868945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.868898 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f00eb47b-3c81-4420-a948-45dae9800548-maas-api-service-ca" pod "f00eb47b-3c81-4420-a948-45dae9800548" (UID: "f00eb47b-3c81-4420-a948-45dae9800548"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:20.869695 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.869642 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f00eb47b-3c81-4420-a948-45dae9800548-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:20.869813 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.869695 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19dafab6-6daa-4e31-bc89-3ebd04adffbb-maas-api-service-ca" pod "19dafab6-6daa-4e31-bc89-3ebd04adffbb" (UID: "19dafab6-6daa-4e31-bc89-3ebd04adffbb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:20.873827 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.872310 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19dafab6-6daa-4e31-bc89-3ebd04adffbb-kube-api-access-rwnsh" pod "19dafab6-6daa-4e31-bc89-3ebd04adffbb" (UID: "19dafab6-6daa-4e31-bc89-3ebd04adffbb"). InnerVolumeSpecName "kube-api-access-rwnsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:20.873827 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.872661 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f00eb47b-3c81-4420-a948-45dae9800548-kube-api-access-w9rbf" pod "f00eb47b-3c81-4420-a948-45dae9800548" (UID: "f00eb47b-3c81-4420-a948-45dae9800548"). InnerVolumeSpecName "kube-api-access-w9rbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:20.953166 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.953118 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj"] Oct 30 21:45:20.957791 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.957729 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz"] Oct 30 21:45:20.970978 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.970939 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-56ac74-29889945-pjwdz"] Oct 30 21:45:20.971272 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.971246 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwnsh\" (UniqueName: \"kubernetes.io/projected/19dafab6-6daa-4e31-bc89-3ebd04adffbb-kube-api-access-rwnsh\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:20.971350 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.971278 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9rbf\" (UniqueName: \"kubernetes.io/projected/f00eb47b-3c81-4420-a948-45dae9800548-kube-api-access-w9rbf\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:20.971350 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:20.971290 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19dafab6-6daa-4e31-bc89-3ebd04adffbb-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:21.019813 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.019782 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a03d85d-cd11-4d9a-ae67-29ced159f159" path="/var/lib/kubelet/pods/2a03d85d-cd11-4d9a-ae67-29ced159f159/volumes" Oct 30 21:45:21.020377 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.020360 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0bb7714-22f3-4fe2-887c-71d638b4d61d" path="/var/lib/kubelet/pods/b0bb7714-22f3-4fe2-887c-71d638b4d61d/volumes" Oct 30 21:45:21.020800 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.020786 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c61bf7ef-856d-4d2e-b4ca-402028fabd8f" path="/var/lib/kubelet/pods/c61bf7ef-856d-4d2e-b4ca-402028fabd8f/volumes" Oct 30 21:45:21.021538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.021519 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3" path="/var/lib/kubelet/pods/e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3/volumes" Oct 30 21:45:21.022163 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.022147 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p"] Oct 30 21:45:21.022443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.022421 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4" Oct 30 21:45:21.050304 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.034878 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-58fea9-29889945-rzg4p"] Oct 30 21:45:21.072365 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.072327 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30def38d-8a67-4281-9031-c6f7f7c96bc5-kube-api-access-qnlk9\" (UniqueName: \"kubernetes.io/projected/30def38d-8a67-4281-9031-c6f7f7c96bc5-kube-api-access-qnlk9\") pod \"30def38d-8a67-4281-9031-c6f7f7c96bc5\" (UID: \"30def38d-8a67-4281-9031-c6f7f7c96bc5\") " Oct 30 21:45:21.072582 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.072558 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30def38d-8a67-4281-9031-c6f7f7c96bc5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30def38d-8a67-4281-9031-c6f7f7c96bc5-maas-api-service-ca\") pod \"30def38d-8a67-4281-9031-c6f7f7c96bc5\" (UID: \"30def38d-8a67-4281-9031-c6f7f7c96bc5\") " Oct 30 21:45:21.074482 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.074456 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30def38d-8a67-4281-9031-c6f7f7c96bc5-maas-api-service-ca" pod "30def38d-8a67-4281-9031-c6f7f7c96bc5" (UID: "30def38d-8a67-4281-9031-c6f7f7c96bc5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:21.076697 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.076668 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30def38d-8a67-4281-9031-c6f7f7c96bc5-kube-api-access-qnlk9" pod "30def38d-8a67-4281-9031-c6f7f7c96bc5" (UID: "30def38d-8a67-4281-9031-c6f7f7c96bc5"). InnerVolumeSpecName "kube-api-access-qnlk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:21.174965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.174929 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30def38d-8a67-4281-9031-c6f7f7c96bc5-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:21.174965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.174960 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnlk9\" (UniqueName: \"kubernetes.io/projected/30def38d-8a67-4281-9031-c6f7f7c96bc5-kube-api-access-qnlk9\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:21.209989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.209901 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm"] Oct 30 21:45:21.215060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.215012 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-9cd83f-29889945-7t6pm"] Oct 30 21:45:21.533790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.533758 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn" Oct 30 21:45:21.533967 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.533748 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn" event={"ID":"19dafab6-6daa-4e31-bc89-3ebd04adffbb","Type":"ContainerDied","Data":"08e0e754f285f6f329b04f7dd25542fb0cf7482c6d52d89e9053e093beb85413"} Oct 30 21:45:21.533967 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.533940 4926 scope.go:122] "RemoveContainer" containerID="fbef41a795a4bf5f846dee6dc93fe7b73f8611bf0a348c46b5d01d8f07add044" Oct 30 21:45:21.540895 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.540867 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t" Oct 30 21:45:21.541076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.540858 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t" event={"ID":"f00eb47b-3c81-4420-a948-45dae9800548","Type":"ContainerDied","Data":"188a95ec77cb36a21021d3d84011140bf1a5d7f8278071fd886e7290dd0f23bd"} Oct 30 21:45:21.558043 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.557988 4926 scope.go:122] "RemoveContainer" containerID="30a3c16ee5295ab014e60d3679af070ac7bde249291a03137a3e7de90d800288" Oct 30 21:45:21.561213 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.561181 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4" event={"ID":"30def38d-8a67-4281-9031-c6f7f7c96bc5","Type":"ContainerDied","Data":"d54d5710d71fde2ba37668627a22cf8cb780578b64660b0f7ef8e8af69ef2a79"} Oct 30 21:45:21.561213 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.561215 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4" Oct 30 21:45:21.572310 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.572199 4926 scope.go:122] "RemoveContainer" containerID="8e3d8f79a606bc524d6583edaebaddc69e010aa68bf8e97b04040f6b176b092c" Oct 30 21:45:21.625494 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.625460 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t"] Oct 30 21:45:21.630610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.630575 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-e11bef-29889945-cc52t"] Oct 30 21:45:21.668839 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.668796 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn"] Oct 30 21:45:21.677084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.674095 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4"] Oct 30 21:45:21.679575 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.679548 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-2cda44-29889945-8mjpn"] Oct 30 21:45:21.736263 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.736225 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95"] Oct 30 21:45:21.755202 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.754787 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj" Oct 30 21:45:21.790862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.789553 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4"] Oct 30 21:45:21.800866 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.797889 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-66cc54-29889945-qgcg4"] Oct 30 21:45:21.853626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.850441 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz"] Oct 30 21:45:21.897355 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.897321 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/657d2744-b23d-4dbb-a75d-23fea5f9a286-kube-api-access-rm4gl\" (UniqueName: \"kubernetes.io/projected/657d2744-b23d-4dbb-a75d-23fea5f9a286-kube-api-access-rm4gl\") pod \"657d2744-b23d-4dbb-a75d-23fea5f9a286\" (UID: \"657d2744-b23d-4dbb-a75d-23fea5f9a286\") " Oct 30 21:45:21.897752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.897734 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/657d2744-b23d-4dbb-a75d-23fea5f9a286-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/657d2744-b23d-4dbb-a75d-23fea5f9a286-maas-api-service-ca\") pod \"657d2744-b23d-4dbb-a75d-23fea5f9a286\" (UID: \"657d2744-b23d-4dbb-a75d-23fea5f9a286\") " Oct 30 21:45:21.902368 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.902005 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/657d2744-b23d-4dbb-a75d-23fea5f9a286-maas-api-service-ca" pod "657d2744-b23d-4dbb-a75d-23fea5f9a286" (UID: "657d2744-b23d-4dbb-a75d-23fea5f9a286"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:21.912579 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.910781 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2"] Oct 30 21:45:21.914245 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.914208 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657d2744-b23d-4dbb-a75d-23fea5f9a286-kube-api-access-rm4gl" pod "657d2744-b23d-4dbb-a75d-23fea5f9a286" (UID: "657d2744-b23d-4dbb-a75d-23fea5f9a286"). InnerVolumeSpecName "kube-api-access-rm4gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:21.915308 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.915216 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4" Oct 30 21:45:22.003077 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.998831 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f27f31-e62c-482e-bfba-10c2250410a4-kube-api-access-fgmwx\" (UniqueName: \"kubernetes.io/projected/d0f27f31-e62c-482e-bfba-10c2250410a4-kube-api-access-fgmwx\") pod \"d0f27f31-e62c-482e-bfba-10c2250410a4\" (UID: \"d0f27f31-e62c-482e-bfba-10c2250410a4\") " Oct 30 21:45:22.003077 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.999123 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f27f31-e62c-482e-bfba-10c2250410a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f27f31-e62c-482e-bfba-10c2250410a4-maas-api-service-ca\") pod \"d0f27f31-e62c-482e-bfba-10c2250410a4\" (UID: \"d0f27f31-e62c-482e-bfba-10c2250410a4\") " Oct 30 21:45:22.003077 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.999618 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/657d2744-b23d-4dbb-a75d-23fea5f9a286-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:22.003077 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:21.999635 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm4gl\" (UniqueName: \"kubernetes.io/projected/657d2744-b23d-4dbb-a75d-23fea5f9a286-kube-api-access-rm4gl\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:22.003077 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.000042 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f27f31-e62c-482e-bfba-10c2250410a4-maas-api-service-ca" pod "d0f27f31-e62c-482e-bfba-10c2250410a4" (UID: "d0f27f31-e62c-482e-bfba-10c2250410a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:22.004477 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.004431 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f27f31-e62c-482e-bfba-10c2250410a4-kube-api-access-fgmwx" pod "d0f27f31-e62c-482e-bfba-10c2250410a4" (UID: "d0f27f31-e62c-482e-bfba-10c2250410a4"). InnerVolumeSpecName "kube-api-access-fgmwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:22.021026 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.021000 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95" Oct 30 21:45:22.102063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.096332 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz" Oct 30 21:45:22.116670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.113576 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd055827-04e0-4946-bf79-b8e59e81fa21-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd055827-04e0-4946-bf79-b8e59e81fa21-maas-api-service-ca\") pod \"bd055827-04e0-4946-bf79-b8e59e81fa21\" (UID: \"bd055827-04e0-4946-bf79-b8e59e81fa21\") " Oct 30 21:45:22.116670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.113716 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd055827-04e0-4946-bf79-b8e59e81fa21-kube-api-access-mwqf6\" (UniqueName: \"kubernetes.io/projected/bd055827-04e0-4946-bf79-b8e59e81fa21-kube-api-access-mwqf6\") pod \"bd055827-04e0-4946-bf79-b8e59e81fa21\" (UID: \"bd055827-04e0-4946-bf79-b8e59e81fa21\") " Oct 30 21:45:22.116670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.114215 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0f27f31-e62c-482e-bfba-10c2250410a4-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:22.116670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.114232 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgmwx\" (UniqueName: \"kubernetes.io/projected/d0f27f31-e62c-482e-bfba-10c2250410a4-kube-api-access-fgmwx\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:22.116670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.114342 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd055827-04e0-4946-bf79-b8e59e81fa21-maas-api-service-ca" pod "bd055827-04e0-4946-bf79-b8e59e81fa21" (UID: "bd055827-04e0-4946-bf79-b8e59e81fa21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:22.117553 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.117523 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd055827-04e0-4946-bf79-b8e59e81fa21-kube-api-access-mwqf6" pod "bd055827-04e0-4946-bf79-b8e59e81fa21" (UID: "bd055827-04e0-4946-bf79-b8e59e81fa21"). InnerVolumeSpecName "kube-api-access-mwqf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:22.206201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.206173 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2" Oct 30 21:45:22.215307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.215277 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543-maas-api-service-ca\") pod \"42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543\" (UID: \"42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543\") " Oct 30 21:45:22.215520 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.215506 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543-kube-api-access-b7g67\" (UniqueName: \"kubernetes.io/projected/42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543-kube-api-access-b7g67\") pod \"42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543\" (UID: \"42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543\") " Oct 30 21:45:22.215642 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.215616 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543-maas-api-service-ca" pod "42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543" (UID: "42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:22.215930 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.215914 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:22.215995 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.215932 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd055827-04e0-4946-bf79-b8e59e81fa21-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:22.215995 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.215942 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwqf6\" (UniqueName: \"kubernetes.io/projected/bd055827-04e0-4946-bf79-b8e59e81fa21-kube-api-access-mwqf6\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:22.217669 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.217647 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543-kube-api-access-b7g67" pod "42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543" (UID: "42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543"). InnerVolumeSpecName "kube-api-access-b7g67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:22.316673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.316579 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/547cf37c-665f-4c4b-ab6f-06b5422b9cc9-kube-api-access-hwqwp\" (UniqueName: \"kubernetes.io/projected/547cf37c-665f-4c4b-ab6f-06b5422b9cc9-kube-api-access-hwqwp\") pod \"547cf37c-665f-4c4b-ab6f-06b5422b9cc9\" (UID: \"547cf37c-665f-4c4b-ab6f-06b5422b9cc9\") " Oct 30 21:45:22.316989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.316960 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/547cf37c-665f-4c4b-ab6f-06b5422b9cc9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/547cf37c-665f-4c4b-ab6f-06b5422b9cc9-maas-api-service-ca\") pod \"547cf37c-665f-4c4b-ab6f-06b5422b9cc9\" (UID: \"547cf37c-665f-4c4b-ab6f-06b5422b9cc9\") " Oct 30 21:45:22.317371 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.317346 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/547cf37c-665f-4c4b-ab6f-06b5422b9cc9-maas-api-service-ca" pod "547cf37c-665f-4c4b-ab6f-06b5422b9cc9" (UID: "547cf37c-665f-4c4b-ab6f-06b5422b9cc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:22.317614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.317599 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7g67\" (UniqueName: \"kubernetes.io/projected/42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543-kube-api-access-b7g67\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:22.317689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.317615 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/547cf37c-665f-4c4b-ab6f-06b5422b9cc9-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:22.318843 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.318820 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/547cf37c-665f-4c4b-ab6f-06b5422b9cc9-kube-api-access-hwqwp" pod "547cf37c-665f-4c4b-ab6f-06b5422b9cc9" (UID: "547cf37c-665f-4c4b-ab6f-06b5422b9cc9"). InnerVolumeSpecName "kube-api-access-hwqwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:22.419234 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.419191 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwqwp\" (UniqueName: \"kubernetes.io/projected/547cf37c-665f-4c4b-ab6f-06b5422b9cc9-kube-api-access-hwqwp\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:45:22.572511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.572410 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj" Oct 30 21:45:22.572860 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.572407 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj" event={"ID":"657d2744-b23d-4dbb-a75d-23fea5f9a286","Type":"ContainerDied","Data":"399b43ed9c492f3f8f44f0687080aba39d1f1f192111d246f3a9a10134255bb7"} Oct 30 21:45:22.572860 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.572648 4926 scope.go:122] "RemoveContainer" containerID="2f40956b289a75d382627b11846ce4f0771913920a04630e467819958797b4a5" Oct 30 21:45:22.575937 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.575917 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz" Oct 30 21:45:22.575937 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.575926 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz" event={"ID":"42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543","Type":"ContainerDied","Data":"95b27353e9337829985abb9771d5d02a29c075a3e10a7986af9b43dedfb7d2a4"} Oct 30 21:45:22.577446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.577427 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95" Oct 30 21:45:22.577569 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.577471 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95" event={"ID":"bd055827-04e0-4946-bf79-b8e59e81fa21","Type":"ContainerDied","Data":"6704aa357dcd1ed39c18c053dd76e3982bd195fd8480177cb2fb2f4f062fb533"} Oct 30 21:45:22.588360 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.588319 4926 scope.go:122] "RemoveContainer" containerID="0ac7ae4197c6e7ae46dca10985e9222f3d7974895693958d23c38286ca996062" Oct 30 21:45:22.590284 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.589447 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4" Oct 30 21:45:22.590284 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.590094 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4" event={"ID":"d0f27f31-e62c-482e-bfba-10c2250410a4","Type":"ContainerDied","Data":"7075ae15bc1a57ebea482c6f2ae02879079f9a71e334199a03c0f37f0920b608"} Oct 30 21:45:22.592777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.592501 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2" event={"ID":"547cf37c-665f-4c4b-ab6f-06b5422b9cc9","Type":"ContainerDied","Data":"bb25c12a1a0337156145519c0f4c4a1e1ded9373017853e9d6ff25a9153b3ef6"} Oct 30 21:45:22.592777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.592590 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2" Oct 30 21:45:22.604076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.604019 4926 scope.go:122] "RemoveContainer" containerID="1ae1846caf046ce5514f1f5b1477aa55a1c1ed18aba32f9f3c698217c88cfddf" Oct 30 21:45:22.616418 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.616393 4926 scope.go:122] "RemoveContainer" containerID="89211d446c4e209ef2336624e4b8b3e530033fd9d9556d46b44a626bccf54d03" Oct 30 21:45:22.627525 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.627494 4926 scope.go:122] "RemoveContainer" containerID="94bab3925c86108979fdb33accb61322b58c2e85f15f042941f4de744585b028" Oct 30 21:45:22.691679 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.691633 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2"] Oct 30 21:45:22.701943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.701908 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-908469-29889945-cskk2"] Oct 30 21:45:22.734325 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.734279 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4"] Oct 30 21:45:22.738559 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.738526 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-ad73a4-29889945-4qmk4"] Oct 30 21:45:22.789812 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.789773 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95"] Oct 30 21:45:22.794192 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.794157 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-5sb95"] Oct 30 21:45:22.809817 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.809780 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj"] Oct 30 21:45:22.813506 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.813472 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-839163-29889945-9nhnj"] Oct 30 21:45:22.845983 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.845893 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz"] Oct 30 21:45:22.854948 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:22.854911 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-43c070-29889945-kswqz"] Oct 30 21:45:23.018541 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:23.018498 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19dafab6-6daa-4e31-bc89-3ebd04adffbb" path="/var/lib/kubelet/pods/19dafab6-6daa-4e31-bc89-3ebd04adffbb/volumes" Oct 30 21:45:23.019148 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:23.019103 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="205a1101-2a99-4800-84d8-56a39d3690b6" path="/var/lib/kubelet/pods/205a1101-2a99-4800-84d8-56a39d3690b6/volumes" Oct 30 21:45:23.019579 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:23.019562 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30def38d-8a67-4281-9031-c6f7f7c96bc5" path="/var/lib/kubelet/pods/30def38d-8a67-4281-9031-c6f7f7c96bc5/volumes" Oct 30 21:45:23.020351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:23.020333 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543" path="/var/lib/kubelet/pods/42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543/volumes" Oct 30 21:45:23.020768 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:23.020753 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="547cf37c-665f-4c4b-ab6f-06b5422b9cc9" path="/var/lib/kubelet/pods/547cf37c-665f-4c4b-ab6f-06b5422b9cc9/volumes" Oct 30 21:45:23.021193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:23.021178 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="657d2744-b23d-4dbb-a75d-23fea5f9a286" path="/var/lib/kubelet/pods/657d2744-b23d-4dbb-a75d-23fea5f9a286/volumes" Oct 30 21:45:23.021852 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:23.021834 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd055827-04e0-4946-bf79-b8e59e81fa21" path="/var/lib/kubelet/pods/bd055827-04e0-4946-bf79-b8e59e81fa21/volumes" Oct 30 21:45:23.022252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:23.022237 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c51dea80-dea4-4195-8c0a-8f28ea265669" path="/var/lib/kubelet/pods/c51dea80-dea4-4195-8c0a-8f28ea265669/volumes" Oct 30 21:45:23.022639 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:23.022625 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f27f31-e62c-482e-bfba-10c2250410a4" path="/var/lib/kubelet/pods/d0f27f31-e62c-482e-bfba-10c2250410a4/volumes" Oct 30 21:45:23.023347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:23.023330 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f00eb47b-3c81-4420-a948-45dae9800548" path="/var/lib/kubelet/pods/f00eb47b-3c81-4420-a948-45dae9800548/volumes" Oct 30 21:45:28.794411 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:28.794359 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:28.812967 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:28.812930 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:29.683263 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:29.683220 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:45:31.365700 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.365658 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44"] Oct 30 21:45:31.366411 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366391 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c51dea80-dea4-4195-8c0a-8f28ea265669" containerName="cleanup" Oct 30 21:45:31.366459 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366414 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51dea80-dea4-4195-8c0a-8f28ea265669" containerName="cleanup" Oct 30 21:45:31.366459 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366432 4926 cpu_manager.go:414] "containerMap: removing container" podUID="2a03d85d-cd11-4d9a-ae67-29ced159f159" containerName="cleanup" Oct 30 21:45:31.366459 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366439 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a03d85d-cd11-4d9a-ae67-29ced159f159" containerName="cleanup" Oct 30 21:45:31.366561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366462 4926 cpu_manager.go:414] "containerMap: removing container" podUID="bd055827-04e0-4946-bf79-b8e59e81fa21" containerName="cleanup" Oct 30 21:45:31.366561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366470 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd055827-04e0-4946-bf79-b8e59e81fa21" containerName="cleanup" Oct 30 21:45:31.366561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366480 4926 cpu_manager.go:414] "containerMap: removing container" podUID="2a03d85d-cd11-4d9a-ae67-29ced159f159" containerName="cleanup" Oct 30 21:45:31.366561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366487 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a03d85d-cd11-4d9a-ae67-29ced159f159" containerName="cleanup" Oct 30 21:45:31.366561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366499 4926 cpu_manager.go:414] "containerMap: removing container" podUID="205a1101-2a99-4800-84d8-56a39d3690b6" containerName="cleanup" Oct 30 21:45:31.366561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366507 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="205a1101-2a99-4800-84d8-56a39d3690b6" containerName="cleanup" Oct 30 21:45:31.366561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366516 4926 cpu_manager.go:414] "containerMap: removing container" podUID="30def38d-8a67-4281-9031-c6f7f7c96bc5" containerName="cleanup" Oct 30 21:45:31.366561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366522 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30def38d-8a67-4281-9031-c6f7f7c96bc5" containerName="cleanup" Oct 30 21:45:31.366561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366532 4926 cpu_manager.go:414] "containerMap: removing container" podUID="b0bb7714-22f3-4fe2-887c-71d638b4d61d" containerName="cleanup" Oct 30 21:45:31.366561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366539 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bb7714-22f3-4fe2-887c-71d638b4d61d" containerName="cleanup" Oct 30 21:45:31.366561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366550 4926 cpu_manager.go:414] "containerMap: removing container" podUID="bd055827-04e0-4946-bf79-b8e59e81fa21" containerName="cleanup" Oct 30 21:45:31.366561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366556 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd055827-04e0-4946-bf79-b8e59e81fa21" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366568 4926 cpu_manager.go:414] "containerMap: removing container" podUID="547cf37c-665f-4c4b-ab6f-06b5422b9cc9" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366574 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="547cf37c-665f-4c4b-ab6f-06b5422b9cc9" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366584 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d0f27f31-e62c-482e-bfba-10c2250410a4" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366591 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f27f31-e62c-482e-bfba-10c2250410a4" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366601 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c51dea80-dea4-4195-8c0a-8f28ea265669" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366608 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51dea80-dea4-4195-8c0a-8f28ea265669" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366618 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c61bf7ef-856d-4d2e-b4ca-402028fabd8f" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366627 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61bf7ef-856d-4d2e-b4ca-402028fabd8f" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366636 4926 cpu_manager.go:414] "containerMap: removing container" podUID="f00eb47b-3c81-4420-a948-45dae9800548" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366643 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f00eb47b-3c81-4420-a948-45dae9800548" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366652 4926 cpu_manager.go:414] "containerMap: removing container" podUID="657d2744-b23d-4dbb-a75d-23fea5f9a286" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366659 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657d2744-b23d-4dbb-a75d-23fea5f9a286" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366673 4926 cpu_manager.go:414] "containerMap: removing container" podUID="e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366680 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366690 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c61bf7ef-856d-4d2e-b4ca-402028fabd8f" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366697 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61bf7ef-856d-4d2e-b4ca-402028fabd8f" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366709 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d0f27f31-e62c-482e-bfba-10c2250410a4" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366715 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f27f31-e62c-482e-bfba-10c2250410a4" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366726 4926 cpu_manager.go:414] "containerMap: removing container" podUID="19dafab6-6daa-4e31-bc89-3ebd04adffbb" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366732 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19dafab6-6daa-4e31-bc89-3ebd04adffbb" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366755 4926 cpu_manager.go:414] "containerMap: removing container" podUID="547cf37c-665f-4c4b-ab6f-06b5422b9cc9" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366762 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="547cf37c-665f-4c4b-ab6f-06b5422b9cc9" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366770 4926 cpu_manager.go:414] "containerMap: removing container" podUID="30def38d-8a67-4281-9031-c6f7f7c96bc5" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366780 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30def38d-8a67-4281-9031-c6f7f7c96bc5" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366793 4926 cpu_manager.go:414] "containerMap: removing container" podUID="42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366800 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366813 4926 cpu_manager.go:414] "containerMap: removing container" podUID="19dafab6-6daa-4e31-bc89-3ebd04adffbb" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366820 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19dafab6-6daa-4e31-bc89-3ebd04adffbb" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366830 4926 cpu_manager.go:414] "containerMap: removing container" podUID="205a1101-2a99-4800-84d8-56a39d3690b6" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366837 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="205a1101-2a99-4800-84d8-56a39d3690b6" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366846 4926 cpu_manager.go:414] "containerMap: removing container" podUID="30def38d-8a67-4281-9031-c6f7f7c96bc5" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366853 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30def38d-8a67-4281-9031-c6f7f7c96bc5" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366870 4926 cpu_manager.go:414] "containerMap: removing container" podUID="e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366876 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366888 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d0f27f31-e62c-482e-bfba-10c2250410a4" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366894 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f27f31-e62c-482e-bfba-10c2250410a4" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366909 4926 cpu_manager.go:414] "containerMap: removing container" podUID="f00eb47b-3c81-4420-a948-45dae9800548" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366916 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f00eb47b-3c81-4420-a948-45dae9800548" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366931 4926 cpu_manager.go:414] "containerMap: removing container" podUID="e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366938 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366950 4926 cpu_manager.go:414] "containerMap: removing container" podUID="42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543" containerName="cleanup" Oct 30 21:45:31.366945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366959 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366973 4926 cpu_manager.go:414] "containerMap: removing container" podUID="19dafab6-6daa-4e31-bc89-3ebd04adffbb" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366979 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19dafab6-6daa-4e31-bc89-3ebd04adffbb" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366990 4926 cpu_manager.go:414] "containerMap: removing container" podUID="657d2744-b23d-4dbb-a75d-23fea5f9a286" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.366996 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657d2744-b23d-4dbb-a75d-23fea5f9a286" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367013 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c61bf7ef-856d-4d2e-b4ca-402028fabd8f" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367020 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61bf7ef-856d-4d2e-b4ca-402028fabd8f" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367045 4926 cpu_manager.go:414] "containerMap: removing container" podUID="b0bb7714-22f3-4fe2-887c-71d638b4d61d" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367052 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bb7714-22f3-4fe2-887c-71d638b4d61d" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367069 4926 cpu_manager.go:414] "containerMap: removing container" podUID="657d2744-b23d-4dbb-a75d-23fea5f9a286" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367076 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657d2744-b23d-4dbb-a75d-23fea5f9a286" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367238 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="19dafab6-6daa-4e31-bc89-3ebd04adffbb" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367249 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="657d2744-b23d-4dbb-a75d-23fea5f9a286" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367258 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="205a1101-2a99-4800-84d8-56a39d3690b6" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367267 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="547cf37c-665f-4c4b-ab6f-06b5422b9cc9" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367276 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61bf7ef-856d-4d2e-b4ca-402028fabd8f" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367286 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bb7714-22f3-4fe2-887c-71d638b4d61d" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367295 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367305 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="657d2744-b23d-4dbb-a75d-23fea5f9a286" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367313 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a03d85d-cd11-4d9a-ae67-29ced159f159" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367323 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51dea80-dea4-4195-8c0a-8f28ea265669" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367331 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a03d85d-cd11-4d9a-ae67-29ced159f159" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367342 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd055827-04e0-4946-bf79-b8e59e81fa21" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367351 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a03d85d-cd11-4d9a-ae67-29ced159f159" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367362 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="30def38d-8a67-4281-9031-c6f7f7c96bc5" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367370 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bb7714-22f3-4fe2-887c-71d638b4d61d" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367379 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd055827-04e0-4946-bf79-b8e59e81fa21" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367388 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f27f31-e62c-482e-bfba-10c2250410a4" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367396 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61bf7ef-856d-4d2e-b4ca-402028fabd8f" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367406 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd055827-04e0-4946-bf79-b8e59e81fa21" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367417 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="f00eb47b-3c81-4420-a948-45dae9800548" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367424 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367432 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61bf7ef-856d-4d2e-b4ca-402028fabd8f" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367441 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f27f31-e62c-482e-bfba-10c2250410a4" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367451 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="19dafab6-6daa-4e31-bc89-3ebd04adffbb" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367462 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="547cf37c-665f-4c4b-ab6f-06b5422b9cc9" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367470 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="30def38d-8a67-4281-9031-c6f7f7c96bc5" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367478 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="f00eb47b-3c81-4420-a948-45dae9800548" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367488 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="205a1101-2a99-4800-84d8-56a39d3690b6" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367495 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="30def38d-8a67-4281-9031-c6f7f7c96bc5" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367504 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367512 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51dea80-dea4-4195-8c0a-8f28ea265669" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367521 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f27f31-e62c-482e-bfba-10c2250410a4" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367531 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12d6f2e-ff5b-48f7-bd68-8e2a697f2db3" containerName="cleanup" Oct 30 21:45:31.368799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.367540 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543" containerName="cleanup" Oct 30 21:45:31.377594 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.377563 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.397752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.397720 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44"] Oct 30 21:45:31.413632 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.413595 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-203209-openshift-default-dockercfg-6x8wp\"" Oct 30 21:45:31.462387 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.462341 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-workload-certs\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.462592 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.462397 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istiod-ca-cert\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.462592 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.462507 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psg9n\" (UniqueName: \"kubernetes.io/projected/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-kube-api-access-psg9n\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.462710 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.462689 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-data\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.462781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.462764 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-token\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.462825 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.462800 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-podinfo\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.462934 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.462915 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-credential-socket\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.463009 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.462979 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-envoy\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.463134 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.463098 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-workload-socket\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.568212 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.567907 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-token\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.568212 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.567971 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-podinfo\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.568212 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.568063 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-credential-socket\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.568212 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.568116 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-envoy\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.568212 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.568155 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-workload-socket\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.568717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.568270 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-workload-certs\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.568717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.568296 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istiod-ca-cert\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.568717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.568336 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psg9n\" (UniqueName: \"kubernetes.io/projected/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-kube-api-access-psg9n\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.568717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.568460 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-data\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.573053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.568959 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-data\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.595149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.585457 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-workload-socket\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.595149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.585719 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-credential-socket\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.595149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.586736 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istiod-ca-cert\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.595149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.586982 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-workload-certs\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.613095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.606826 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-token\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.613095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.607449 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-podinfo\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.613095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.607720 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-envoy\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.652099 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.651314 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psg9n\" (UniqueName: \"kubernetes.io/projected/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-kube-api-access-psg9n\") pod \"e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:31.698408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:31.698371 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:32.093770 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:32.093733 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44"] Oct 30 21:45:32.102240 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:45:32.101366 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50bd09f8_bbc4_41f3_ae8f_f517e50a78b5.slice/crio-9db9ef35a5e485e1fa3b979a31629be7c3306146419b8e4ad33dfe0ed128a79b WatchSource:0}: Error finding container 9db9ef35a5e485e1fa3b979a31629be7c3306146419b8e4ad33dfe0ed128a79b: Status 404 returned error can't find the container with id 9db9ef35a5e485e1fa3b979a31629be7c3306146419b8e4ad33dfe0ed128a79b Oct 30 21:45:32.104774 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:32.104728 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:32.104888 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:32.104810 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:32.104888 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:32.104848 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:45:32.715859 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:32.715814 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" event={"ID":"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5","Type":"ContainerStarted","Data":"2043dd192beaa7e842e844f63a0681d3dee2910e3ca5150a364a7e4f4a2853fb"} Oct 30 21:45:32.716368 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:32.715882 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" event={"ID":"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5","Type":"ContainerStarted","Data":"9db9ef35a5e485e1fa3b979a31629be7c3306146419b8e4ad33dfe0ed128a79b"} Oct 30 21:45:32.779350 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:32.779283 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" podStartSLOduration=1.779266535 podStartE2EDuration="1.779266535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:32.778963438 +0000 UTC m=+2719.900037186" watchObservedRunningTime="2026-10-30 21:45:32.779266535 +0000 UTC m=+2719.900340282" Oct 30 21:45:33.702194 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:33.702158 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:33.710082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:33.710046 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:33.725001 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:33.724968 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:33.726627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:33.726602 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:45:36.288976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.288940 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj"] Oct 30 21:45:36.289583 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289565 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c51dea80-dea4-4195-8c0a-8f28ea265669" containerName="cleanup" Oct 30 21:45:36.289626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289584 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51dea80-dea4-4195-8c0a-8f28ea265669" containerName="cleanup" Oct 30 21:45:36.289626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289606 4926 cpu_manager.go:414] "containerMap: removing container" podUID="547cf37c-665f-4c4b-ab6f-06b5422b9cc9" containerName="cleanup" Oct 30 21:45:36.289626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289612 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="547cf37c-665f-4c4b-ab6f-06b5422b9cc9" containerName="cleanup" Oct 30 21:45:36.289740 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289627 4926 cpu_manager.go:414] "containerMap: removing container" podUID="f00eb47b-3c81-4420-a948-45dae9800548" containerName="cleanup" Oct 30 21:45:36.289740 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289632 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f00eb47b-3c81-4420-a948-45dae9800548" containerName="cleanup" Oct 30 21:45:36.289740 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289644 4926 cpu_manager.go:414] "containerMap: removing container" podUID="42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543" containerName="cleanup" Oct 30 21:45:36.289740 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289649 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543" containerName="cleanup" Oct 30 21:45:36.289740 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289680 4926 cpu_manager.go:414] "containerMap: removing container" podUID="2a03d85d-cd11-4d9a-ae67-29ced159f159" containerName="cleanup" Oct 30 21:45:36.289740 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289688 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a03d85d-cd11-4d9a-ae67-29ced159f159" containerName="cleanup" Oct 30 21:45:36.289740 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289697 4926 cpu_manager.go:414] "containerMap: removing container" podUID="205a1101-2a99-4800-84d8-56a39d3690b6" containerName="cleanup" Oct 30 21:45:36.289740 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289703 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="205a1101-2a99-4800-84d8-56a39d3690b6" containerName="cleanup" Oct 30 21:45:36.289740 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289711 4926 cpu_manager.go:414] "containerMap: removing container" podUID="b0bb7714-22f3-4fe2-887c-71d638b4d61d" containerName="cleanup" Oct 30 21:45:36.289740 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289715 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bb7714-22f3-4fe2-887c-71d638b4d61d" containerName="cleanup" Oct 30 21:45:36.289740 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289722 4926 cpu_manager.go:414] "containerMap: removing container" podUID="bd055827-04e0-4946-bf79-b8e59e81fa21" containerName="cleanup" Oct 30 21:45:36.289740 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289727 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd055827-04e0-4946-bf79-b8e59e81fa21" containerName="cleanup" Oct 30 21:45:36.290183 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289823 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="19dafab6-6daa-4e31-bc89-3ebd04adffbb" containerName="cleanup" Oct 30 21:45:36.290183 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289834 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="f00eb47b-3c81-4420-a948-45dae9800548" containerName="cleanup" Oct 30 21:45:36.290183 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289843 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51dea80-dea4-4195-8c0a-8f28ea265669" containerName="cleanup" Oct 30 21:45:36.290183 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289855 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="42ef7e75-1cb4-4758-a7e3-5f6c3c6e7543" containerName="cleanup" Oct 30 21:45:36.290183 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289863 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="205a1101-2a99-4800-84d8-56a39d3690b6" containerName="cleanup" Oct 30 21:45:36.290183 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289869 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="547cf37c-665f-4c4b-ab6f-06b5422b9cc9" containerName="cleanup" Oct 30 21:45:36.290183 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289878 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bb7714-22f3-4fe2-887c-71d638b4d61d" containerName="cleanup" Oct 30 21:45:36.290183 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.289888 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="657d2744-b23d-4dbb-a75d-23fea5f9a286" containerName="cleanup" Oct 30 21:45:36.295064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.295022 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" Oct 30 21:45:36.328230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.328195 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj"] Oct 30 21:45:36.345428 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.345383 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58lrv\" (UniqueName: \"kubernetes.io/projected/44921ce3-eab8-4250-84bb-3bf42ac953cb-kube-api-access-58lrv\") pod \"maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj\" (UID: \"44921ce3-eab8-4250-84bb-3bf42ac953cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" Oct 30 21:45:36.345624 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.345603 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44921ce3-eab8-4250-84bb-3bf42ac953cb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj\" (UID: \"44921ce3-eab8-4250-84bb-3bf42ac953cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" Oct 30 21:45:36.345756 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.345733 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44921ce3-eab8-4250-84bb-3bf42ac953cb-metrics-certs\") pod \"maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj\" (UID: \"44921ce3-eab8-4250-84bb-3bf42ac953cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" Oct 30 21:45:36.345906 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.345889 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44921ce3-eab8-4250-84bb-3bf42ac953cb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj\" (UID: \"44921ce3-eab8-4250-84bb-3bf42ac953cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" Oct 30 21:45:36.380672 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.380634 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c3bbba-serving-cert\"" Oct 30 21:45:36.447428 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.447393 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44921ce3-eab8-4250-84bb-3bf42ac953cb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj\" (UID: \"44921ce3-eab8-4250-84bb-3bf42ac953cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" Oct 30 21:45:36.447640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.447463 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58lrv\" (UniqueName: \"kubernetes.io/projected/44921ce3-eab8-4250-84bb-3bf42ac953cb-kube-api-access-58lrv\") pod \"maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj\" (UID: \"44921ce3-eab8-4250-84bb-3bf42ac953cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" Oct 30 21:45:36.447640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.447540 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44921ce3-eab8-4250-84bb-3bf42ac953cb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj\" (UID: \"44921ce3-eab8-4250-84bb-3bf42ac953cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" Oct 30 21:45:36.447640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.447604 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44921ce3-eab8-4250-84bb-3bf42ac953cb-metrics-certs\") pod \"maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj\" (UID: \"44921ce3-eab8-4250-84bb-3bf42ac953cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" Oct 30 21:45:36.448342 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.448311 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44921ce3-eab8-4250-84bb-3bf42ac953cb-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj\" (UID: \"44921ce3-eab8-4250-84bb-3bf42ac953cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" Oct 30 21:45:36.449966 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.449935 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44921ce3-eab8-4250-84bb-3bf42ac953cb-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj\" (UID: \"44921ce3-eab8-4250-84bb-3bf42ac953cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" Oct 30 21:45:36.450081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.449972 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44921ce3-eab8-4250-84bb-3bf42ac953cb-metrics-certs\") pod \"maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj\" (UID: \"44921ce3-eab8-4250-84bb-3bf42ac953cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" Oct 30 21:45:36.469755 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.469717 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58lrv\" (UniqueName: \"kubernetes.io/projected/44921ce3-eab8-4250-84bb-3bf42ac953cb-kube-api-access-58lrv\") pod \"maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj\" (UID: \"44921ce3-eab8-4250-84bb-3bf42ac953cb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" Oct 30 21:45:36.625671 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.625578 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" Oct 30 21:45:36.951517 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:45:36.951484 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44921ce3_eab8_4250_84bb_3bf42ac953cb.slice/crio-f33a376818e869c84bd1765ddeee551c7b46cd351550ba51574db3fec4f579bf WatchSource:0}: Error finding container f33a376818e869c84bd1765ddeee551c7b46cd351550ba51574db3fec4f579bf: Status 404 returned error can't find the container with id f33a376818e869c84bd1765ddeee551c7b46cd351550ba51574db3fec4f579bf Oct 30 21:45:36.951681 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:36.951647 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj"] Oct 30 21:45:37.523069 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:45:37.519390 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:45:37.882087 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:37.881681 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" event={"ID":"44921ce3-eab8-4250-84bb-3bf42ac953cb","Type":"ContainerStarted","Data":"35ebadad087eff4477feacb4db31ba2a32eac12417bdfb9299d8b25d556c6146"} Oct 30 21:45:37.882087 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:37.881780 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" event={"ID":"44921ce3-eab8-4250-84bb-3bf42ac953cb","Type":"ContainerStarted","Data":"f33a376818e869c84bd1765ddeee551c7b46cd351550ba51574db3fec4f579bf"} Oct 30 21:45:37.882832 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:37.882810 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" Oct 30 21:45:37.955568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:37.955502 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" podStartSLOduration=1.594240402 podStartE2EDuration="1.955484575s" totalImagesPullingTime="361.244173ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:37.955239859 +0000 UTC m=+2725.076313604" watchObservedRunningTime="2026-10-30 21:45:37.955484575 +0000 UTC m=+2725.076558323" Oct 30 21:45:43.897906 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:45:43.897871 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" Oct 30 21:46:05.846299 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:05.846202 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg"] Oct 30 21:46:05.857278 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:05.857250 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:05.873874 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:05.873825 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg"] Oct 30 21:46:05.878891 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:05.878857 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-908469-openshift-default-dockercfg-bvlc4\"" Oct 30 21:46:05.955347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:05.955303 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-workload-certs\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:05.955347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:05.955352 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a624ed-4a10-48c0-9a7a-002ebb6114df-istiod-ca-cert\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:05.955627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:05.955370 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-data\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:05.955627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:05.955497 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-token\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:05.955627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:05.955621 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv9kw\" (UniqueName: \"kubernetes.io/projected/73a624ed-4a10-48c0-9a7a-002ebb6114df-kube-api-access-qv9kw\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:05.955792 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:05.955652 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-podinfo\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:05.955857 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:05.955781 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-envoy\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:05.955914 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:05.955899 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-workload-socket\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:05.955970 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:05.955960 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-credential-socket\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:06.057408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.057367 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-workload-certs\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:06.057408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.057412 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a624ed-4a10-48c0-9a7a-002ebb6114df-istiod-ca-cert\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:06.057705 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.057432 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-data\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:06.057705 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.057474 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-token\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:06.057705 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.057659 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv9kw\" (UniqueName: \"kubernetes.io/projected/73a624ed-4a10-48c0-9a7a-002ebb6114df-kube-api-access-qv9kw\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:06.057947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.057918 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-workload-certs\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:06.058010 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.057941 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-data\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:06.058117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.058060 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-podinfo\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:06.058193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.058140 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-envoy\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:06.058256 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.058210 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-workload-socket\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:06.058320 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.058261 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-credential-socket\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:06.058611 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.058586 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-workload-socket\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:06.058751 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.058617 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-credential-socket\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:06.058804 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.058775 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a624ed-4a10-48c0-9a7a-002ebb6114df-istiod-ca-cert\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:06.060630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.060607 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-envoy\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:06.060892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.060867 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-podinfo\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:06.077066 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.076999 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv9kw\" (UniqueName: \"kubernetes.io/projected/73a624ed-4a10-48c0-9a7a-002ebb6114df-kube-api-access-qv9kw\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:06.077202 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.077124 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-token\") pod \"e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:06.175374 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.175336 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:06.503324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.503289 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg"] Oct 30 21:46:06.506688 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:46:06.506653 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a624ed_4a10_48c0_9a7a_002ebb6114df.slice/crio-66bc45417f6d5544fa9ebbed1c0cde02046a09b1f044f17c61f2c3382cc9b244 WatchSource:0}: Error finding container 66bc45417f6d5544fa9ebbed1c0cde02046a09b1f044f17c61f2c3382cc9b244: Status 404 returned error can't find the container with id 66bc45417f6d5544fa9ebbed1c0cde02046a09b1f044f17c61f2c3382cc9b244 Oct 30 21:46:06.510365 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.510319 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:06.510475 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.510420 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:06.510475 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:06.510463 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:07.020344 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.020309 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv"] Oct 30 21:46:07.024830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.024804 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:07.046068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.046017 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv"] Oct 30 21:46:07.050524 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.050484 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-95692f\"/\"sub-test-model-95692f-kserve-self-signed-certs\"" Oct 30 21:46:07.050711 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.050562 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-95692f\"/\"kube-root-ca.crt\"" Oct 30 21:46:07.052322 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.052294 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-95692f\"/\"openshift-service-ca.crt\"" Oct 30 21:46:07.052528 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.052505 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-95692f\"/\"default-dockercfg-462bg\"" Oct 30 21:46:07.075079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.075012 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/875f580c-1467-4e76-9818-0d0154042c3f-tls-certs\") pod \"sub-test-model-95692f-kserve-6dbcb47d55-prlxv\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:07.075292 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.075203 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-model-cache\") pod \"sub-test-model-95692f-kserve-6dbcb47d55-prlxv\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:07.076976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.076941 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-home\") pod \"sub-test-model-95692f-kserve-6dbcb47d55-prlxv\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:07.077146 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.076995 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-tmp-dir\") pod \"sub-test-model-95692f-kserve-6dbcb47d55-prlxv\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:07.077146 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.077019 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-dshm\") pod \"sub-test-model-95692f-kserve-6dbcb47d55-prlxv\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:07.077299 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.077270 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8q59\" (UniqueName: \"kubernetes.io/projected/875f580c-1467-4e76-9818-0d0154042c3f-kube-api-access-m8q59\") pod \"sub-test-model-95692f-kserve-6dbcb47d55-prlxv\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:07.181203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.181153 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/875f580c-1467-4e76-9818-0d0154042c3f-tls-certs\") pod \"sub-test-model-95692f-kserve-6dbcb47d55-prlxv\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:07.181431 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.181260 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-model-cache\") pod \"sub-test-model-95692f-kserve-6dbcb47d55-prlxv\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:07.181431 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.181300 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-home\") pod \"sub-test-model-95692f-kserve-6dbcb47d55-prlxv\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:07.181431 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.181342 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-tmp-dir\") pod \"sub-test-model-95692f-kserve-6dbcb47d55-prlxv\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:07.181431 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.181367 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-dshm\") pod \"sub-test-model-95692f-kserve-6dbcb47d55-prlxv\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:07.181595 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.181437 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8q59\" (UniqueName: \"kubernetes.io/projected/875f580c-1467-4e76-9818-0d0154042c3f-kube-api-access-m8q59\") pod \"sub-test-model-95692f-kserve-6dbcb47d55-prlxv\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:07.182307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.182275 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-model-cache\") pod \"sub-test-model-95692f-kserve-6dbcb47d55-prlxv\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:07.182307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.182298 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-tmp-dir\") pod \"sub-test-model-95692f-kserve-6dbcb47d55-prlxv\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:07.182883 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.182856 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-home\") pod \"sub-test-model-95692f-kserve-6dbcb47d55-prlxv\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:07.216154 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.185599 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/875f580c-1467-4e76-9818-0d0154042c3f-tls-certs\") pod \"sub-test-model-95692f-kserve-6dbcb47d55-prlxv\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:07.216154 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.209355 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8q59\" (UniqueName: \"kubernetes.io/projected/875f580c-1467-4e76-9818-0d0154042c3f-kube-api-access-m8q59\") pod \"sub-test-model-95692f-kserve-6dbcb47d55-prlxv\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:07.216782 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.216716 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-dshm\") pod \"sub-test-model-95692f-kserve-6dbcb47d55-prlxv\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:07.223087 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.223045 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" event={"ID":"73a624ed-4a10-48c0-9a7a-002ebb6114df","Type":"ContainerStarted","Data":"7f92220df6c2c9884681329f95c7a94de88e08fda0c2be135bca9985d8cdfab5"} Oct 30 21:46:07.223217 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.223121 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" event={"ID":"73a624ed-4a10-48c0-9a7a-002ebb6114df","Type":"ContainerStarted","Data":"66bc45417f6d5544fa9ebbed1c0cde02046a09b1f044f17c61f2c3382cc9b244"} Oct 30 21:46:07.272116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.268267 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" podStartSLOduration=2.268247723 podStartE2EDuration="2.268247723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:07.265548865 +0000 UTC m=+2754.386622593" watchObservedRunningTime="2026-10-30 21:46:07.268247723 +0000 UTC m=+2754.389321470" Oct 30 21:46:07.350377 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.350322 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:07.565128 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:46:07.559447 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:46:07.824679 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:07.824640 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv"] Oct 30 21:46:07.824898 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:46:07.824870 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod875f580c_1467_4e76_9818_0d0154042c3f.slice/crio-e1a5605ffbd3c47facd1ab4abb31174cbcda909c12e658e792474c193784250e WatchSource:0}: Error finding container e1a5605ffbd3c47facd1ab4abb31174cbcda909c12e658e792474c193784250e: Status 404 returned error can't find the container with id e1a5605ffbd3c47facd1ab4abb31174cbcda909c12e658e792474c193784250e Oct 30 21:46:08.189059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:08.178682 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:08.189059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:08.178877 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 21:46:08.189059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:08.178926 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" podUID="73a624ed-4a10-48c0-9a7a-002ebb6114df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 21:46:08.232839 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:08.232802 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" event={"ID":"875f580c-1467-4e76-9818-0d0154042c3f","Type":"ContainerStarted","Data":"2830d94cda182db920ce0ad8a29304cd9f7bc5c564c09db17fec346136587c09"} Oct 30 21:46:08.233058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:08.232850 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" event={"ID":"875f580c-1467-4e76-9818-0d0154042c3f","Type":"ContainerStarted","Data":"e1a5605ffbd3c47facd1ab4abb31174cbcda909c12e658e792474c193784250e"} Oct 30 21:46:08.293384 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:08.293312 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" podStartSLOduration=2.29329259 podStartE2EDuration="2.29329259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:08.290371401 +0000 UTC m=+2755.411445160" watchObservedRunningTime="2026-10-30 21:46:08.29329259 +0000 UTC m=+2755.414366338" Oct 30 21:46:08.350711 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:08.350662 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:09.188058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:09.185474 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 21:46:09.188058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:09.185557 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" podUID="73a624ed-4a10-48c0-9a7a-002ebb6114df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 21:46:10.178483 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:10.178438 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 21:46:10.179145 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:10.179104 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" podUID="73a624ed-4a10-48c0-9a7a-002ebb6114df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 21:46:11.187056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:11.180717 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 21:46:11.187056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:11.180821 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" podUID="73a624ed-4a10-48c0-9a7a-002ebb6114df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 21:46:12.180056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:12.176428 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body= Oct 30 21:46:12.180056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:12.176511 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" podUID="73a624ed-4a10-48c0-9a7a-002ebb6114df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" Oct 30 21:46:13.194016 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:13.193967 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:13.194791 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:13.194630 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:13.195616 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:13.195590 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:46:17.351435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:17.351399 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:17.365389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:17.365356 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:18.289498 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:18.289463 4926 scope.go:122] "RemoveContainer" containerID="5f1470e758efb85fc2619ded37d6f9542eebd76345f2aaef5b98844e028334e8" Oct 30 21:46:18.348251 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:18.347898 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:46:23.224020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:23.223977 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt"] Oct 30 21:46:23.257352 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:23.257312 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7"] Oct 30 21:46:28.224320 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:28.224269 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt" podUID="d5aa7f00-cf51-434d-bb02-a14f937f9f8e" containerName="payload-processing" containerID="cri-o://1b55d02ddf3f0b65626fe6f63d23bdee2857803c00869e81e349629a52325dea" gracePeriod=25 Oct 30 21:46:28.258100 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:28.258058 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7" podUID="dd40b019-c8a9-4ceb-ae90-efa4eba7331f" containerName="payload-pre-processing" containerID="cri-o://c17d4d99192af125f2aa4d1ed366e8ba008e1e0430fc41b14a0a119965905b82" gracePeriod=25 Oct 30 21:46:28.460902 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:28.460863 4926 generic.go:356] "Generic (PLEG): container finished" podID="d5aa7f00-cf51-434d-bb02-a14f937f9f8e" containerID="1b55d02ddf3f0b65626fe6f63d23bdee2857803c00869e81e349629a52325dea" exitCode=0 Oct 30 21:46:28.461130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:28.461050 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt" event={"ID":"d5aa7f00-cf51-434d-bb02-a14f937f9f8e","Type":"ContainerDied","Data":"1b55d02ddf3f0b65626fe6f63d23bdee2857803c00869e81e349629a52325dea"} Oct 30 21:46:28.462738 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:28.462708 4926 generic.go:356] "Generic (PLEG): container finished" podID="dd40b019-c8a9-4ceb-ae90-efa4eba7331f" containerID="c17d4d99192af125f2aa4d1ed366e8ba008e1e0430fc41b14a0a119965905b82" exitCode=0 Oct 30 21:46:28.462869 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:28.462823 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7" event={"ID":"dd40b019-c8a9-4ceb-ae90-efa4eba7331f","Type":"ContainerDied","Data":"c17d4d99192af125f2aa4d1ed366e8ba008e1e0430fc41b14a0a119965905b82"} Oct 30 21:46:28.577966 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:28.577929 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt" Oct 30 21:46:28.608585 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:28.608557 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7" Oct 30 21:46:28.680405 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:28.680333 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5aa7f00-cf51-434d-bb02-a14f937f9f8e-kube-api-access-ptqbw\" (UniqueName: \"kubernetes.io/projected/d5aa7f00-cf51-434d-bb02-a14f937f9f8e-kube-api-access-ptqbw\") pod \"d5aa7f00-cf51-434d-bb02-a14f937f9f8e\" (UID: \"d5aa7f00-cf51-434d-bb02-a14f937f9f8e\") " Oct 30 21:46:28.680618 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:28.680428 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5aa7f00-cf51-434d-bb02-a14f937f9f8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5aa7f00-cf51-434d-bb02-a14f937f9f8e-plugins-config-volume\") pod \"d5aa7f00-cf51-434d-bb02-a14f937f9f8e\" (UID: \"d5aa7f00-cf51-434d-bb02-a14f937f9f8e\") " Oct 30 21:46:28.680884 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:28.680854 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5aa7f00-cf51-434d-bb02-a14f937f9f8e-plugins-config-volume" pod "d5aa7f00-cf51-434d-bb02-a14f937f9f8e" (UID: "d5aa7f00-cf51-434d-bb02-a14f937f9f8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:28.681449 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:28.681423 4926 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5aa7f00-cf51-434d-bb02-a14f937f9f8e-plugins-config-volume\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:46:28.683655 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:28.683623 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5aa7f00-cf51-434d-bb02-a14f937f9f8e-kube-api-access-ptqbw" pod "d5aa7f00-cf51-434d-bb02-a14f937f9f8e" (UID: "d5aa7f00-cf51-434d-bb02-a14f937f9f8e"). InnerVolumeSpecName "kube-api-access-ptqbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:28.789856 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:28.788999 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd40b019-c8a9-4ceb-ae90-efa4eba7331f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd40b019-c8a9-4ceb-ae90-efa4eba7331f-plugins-config-volume\") pod \"dd40b019-c8a9-4ceb-ae90-efa4eba7331f\" (UID: \"dd40b019-c8a9-4ceb-ae90-efa4eba7331f\") " Oct 30 21:46:28.789856 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:28.789109 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd40b019-c8a9-4ceb-ae90-efa4eba7331f-kube-api-access-9fmgk\" (UniqueName: \"kubernetes.io/projected/dd40b019-c8a9-4ceb-ae90-efa4eba7331f-kube-api-access-9fmgk\") pod \"dd40b019-c8a9-4ceb-ae90-efa4eba7331f\" (UID: \"dd40b019-c8a9-4ceb-ae90-efa4eba7331f\") " Oct 30 21:46:28.792812 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:28.792671 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd40b019-c8a9-4ceb-ae90-efa4eba7331f-plugins-config-volume" pod "dd40b019-c8a9-4ceb-ae90-efa4eba7331f" (UID: "dd40b019-c8a9-4ceb-ae90-efa4eba7331f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:28.793088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:28.793062 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptqbw\" (UniqueName: \"kubernetes.io/projected/d5aa7f00-cf51-434d-bb02-a14f937f9f8e-kube-api-access-ptqbw\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:46:28.793206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:28.793093 4926 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd40b019-c8a9-4ceb-ae90-efa4eba7331f-plugins-config-volume\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:46:28.795547 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:28.795514 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd40b019-c8a9-4ceb-ae90-efa4eba7331f-kube-api-access-9fmgk" pod "dd40b019-c8a9-4ceb-ae90-efa4eba7331f" (UID: "dd40b019-c8a9-4ceb-ae90-efa4eba7331f"). InnerVolumeSpecName "kube-api-access-9fmgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:28.894360 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:28.894304 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fmgk\" (UniqueName: \"kubernetes.io/projected/dd40b019-c8a9-4ceb-ae90-efa4eba7331f-kube-api-access-9fmgk\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:46:29.034751 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:29.034719 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk"] Oct 30 21:46:29.038429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:29.038404 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8b763b4-mjjhk"] Oct 30 21:46:29.474870 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:29.474833 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt" Oct 30 21:46:29.474870 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:29.474864 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt" event={"ID":"d5aa7f00-cf51-434d-bb02-a14f937f9f8e","Type":"ContainerDied","Data":"b5fd662dc280faa6fa6c36e999eaed94f2c4defa2f26d1066615081e7287e9e7"} Oct 30 21:46:29.475487 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:29.474932 4926 scope.go:122] "RemoveContainer" containerID="1b55d02ddf3f0b65626fe6f63d23bdee2857803c00869e81e349629a52325dea" Oct 30 21:46:29.476898 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:29.476872 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7" event={"ID":"dd40b019-c8a9-4ceb-ae90-efa4eba7331f","Type":"ContainerDied","Data":"2fbd2f5abce63f07600efffbbc6edff935a1fbded95e0eed25c2392c42cb474b"} Oct 30 21:46:29.477025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:29.476971 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7" Oct 30 21:46:29.487955 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:29.487928 4926 scope.go:122] "RemoveContainer" containerID="c17d4d99192af125f2aa4d1ed366e8ba008e1e0430fc41b14a0a119965905b82" Oct 30 21:46:29.526091 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:29.526052 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt"] Oct 30 21:46:29.537193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:29.537163 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c3bbba-55dddc5984-9dxtt"] Oct 30 21:46:29.561146 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:29.561108 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7"] Oct 30 21:46:29.583656 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:29.583616 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c3bbba-6586575dff-9zjb7"] Oct 30 21:46:31.018320 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:31.018287 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22c6c8db-3593-46bc-a7c8-6475696ef182" path="/var/lib/kubelet/pods/22c6c8db-3593-46bc-a7c8-6475696ef182/volumes" Oct 30 21:46:31.018781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:31.018766 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5aa7f00-cf51-434d-bb02-a14f937f9f8e" path="/var/lib/kubelet/pods/d5aa7f00-cf51-434d-bb02-a14f937f9f8e/volumes" Oct 30 21:46:31.019195 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:31.019180 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd40b019-c8a9-4ceb-ae90-efa4eba7331f" path="/var/lib/kubelet/pods/dd40b019-c8a9-4ceb-ae90-efa4eba7331f/volumes" Oct 30 21:46:36.916645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:36.916604 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91171a3c-v5hdk"] Oct 30 21:46:36.917336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:36.917261 4926 cpu_manager.go:414] "containerMap: removing container" podUID="dd40b019-c8a9-4ceb-ae90-efa4eba7331f" containerName="payload-pre-processing" Oct 30 21:46:36.917336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:36.917277 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd40b019-c8a9-4ceb-ae90-efa4eba7331f" containerName="payload-pre-processing" Oct 30 21:46:36.917336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:36.917319 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d5aa7f00-cf51-434d-bb02-a14f937f9f8e" containerName="payload-processing" Oct 30 21:46:36.917336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:36.917324 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5aa7f00-cf51-434d-bb02-a14f937f9f8e" containerName="payload-processing" Oct 30 21:46:36.917534 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:36.917434 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd40b019-c8a9-4ceb-ae90-efa4eba7331f" containerName="payload-pre-processing" Oct 30 21:46:36.917534 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:36.917451 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5aa7f00-cf51-434d-bb02-a14f937f9f8e" containerName="payload-processing" Oct 30 21:46:36.922763 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:36.922739 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91171a3c-v5hdk" Oct 30 21:46:36.938010 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:36.937979 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91171a3c-v5hdk"] Oct 30 21:46:36.943320 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:36.943290 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:46:36.943536 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:36.943403 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:46:37.017580 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:37.017542 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-458qz\" (UniqueName: \"kubernetes.io/projected/d131007a-17b7-4e82-9e38-96b96aed4932-kube-api-access-458qz\") pod \"maas-api-revoke-sub-91171a3c-v5hdk\" (UID: \"d131007a-17b7-4e82-9e38-96b96aed4932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91171a3c-v5hdk" Oct 30 21:46:37.017780 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:37.017605 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d131007a-17b7-4e82-9e38-96b96aed4932-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91171a3c-v5hdk\" (UID: \"d131007a-17b7-4e82-9e38-96b96aed4932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91171a3c-v5hdk" Oct 30 21:46:37.017780 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:37.017773 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d131007a-17b7-4e82-9e38-96b96aed4932-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91171a3c-v5hdk\" (UID: \"d131007a-17b7-4e82-9e38-96b96aed4932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91171a3c-v5hdk" Oct 30 21:46:37.118916 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:37.118873 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-458qz\" (UniqueName: \"kubernetes.io/projected/d131007a-17b7-4e82-9e38-96b96aed4932-kube-api-access-458qz\") pod \"maas-api-revoke-sub-91171a3c-v5hdk\" (UID: \"d131007a-17b7-4e82-9e38-96b96aed4932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91171a3c-v5hdk" Oct 30 21:46:37.119171 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:37.118927 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d131007a-17b7-4e82-9e38-96b96aed4932-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91171a3c-v5hdk\" (UID: \"d131007a-17b7-4e82-9e38-96b96aed4932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91171a3c-v5hdk" Oct 30 21:46:37.119171 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:37.118985 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d131007a-17b7-4e82-9e38-96b96aed4932-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91171a3c-v5hdk\" (UID: \"d131007a-17b7-4e82-9e38-96b96aed4932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91171a3c-v5hdk" Oct 30 21:46:37.119799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:37.119773 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d131007a-17b7-4e82-9e38-96b96aed4932-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91171a3c-v5hdk\" (UID: \"d131007a-17b7-4e82-9e38-96b96aed4932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91171a3c-v5hdk" Oct 30 21:46:37.121641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:37.121618 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d131007a-17b7-4e82-9e38-96b96aed4932-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91171a3c-v5hdk\" (UID: \"d131007a-17b7-4e82-9e38-96b96aed4932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91171a3c-v5hdk" Oct 30 21:46:37.139844 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:37.139816 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-458qz\" (UniqueName: \"kubernetes.io/projected/d131007a-17b7-4e82-9e38-96b96aed4932-kube-api-access-458qz\") pod \"maas-api-revoke-sub-91171a3c-v5hdk\" (UID: \"d131007a-17b7-4e82-9e38-96b96aed4932\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91171a3c-v5hdk" Oct 30 21:46:37.238844 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:37.238757 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91171a3c-v5hdk" Oct 30 21:46:37.487499 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:46:37.487451 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:46:37.551105 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:46:37.551026 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd131007a_17b7_4e82_9e38_96b96aed4932.slice/crio-f7ca79f4a5f0fb4ac8b82966223224dd87219c7b06e88c870c9fd00c8c3552eb WatchSource:0}: Error finding container f7ca79f4a5f0fb4ac8b82966223224dd87219c7b06e88c870c9fd00c8c3552eb: Status 404 returned error can't find the container with id f7ca79f4a5f0fb4ac8b82966223224dd87219c7b06e88c870c9fd00c8c3552eb Oct 30 21:46:37.551598 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:37.551571 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91171a3c-v5hdk"] Oct 30 21:46:38.573243 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:38.573187 4926 generic.go:356] "Generic (PLEG): container finished" podID="d131007a-17b7-4e82-9e38-96b96aed4932" containerID="b9e57c58b2fd1a9686f5caa79bc8c5f66421b8be19ae7118a48f64fb8833ddfc" exitCode=0 Oct 30 21:46:38.573661 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:38.573369 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91171a3c-v5hdk" event={"ID":"d131007a-17b7-4e82-9e38-96b96aed4932","Type":"ContainerDied","Data":"b9e57c58b2fd1a9686f5caa79bc8c5f66421b8be19ae7118a48f64fb8833ddfc"} Oct 30 21:46:38.573661 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:38.573450 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91171a3c-v5hdk" event={"ID":"d131007a-17b7-4e82-9e38-96b96aed4932","Type":"ContainerStarted","Data":"f7ca79f4a5f0fb4ac8b82966223224dd87219c7b06e88c870c9fd00c8c3552eb"} Oct 30 21:46:39.062818 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:39.062775 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6"] Oct 30 21:46:39.067291 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:39.067235 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7bf0b5f0-srgl6"] Oct 30 21:46:39.766706 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:39.766668 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91171a3c-v5hdk" Oct 30 21:46:39.857249 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:39.857145 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d131007a-17b7-4e82-9e38-96b96aed4932-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d131007a-17b7-4e82-9e38-96b96aed4932-maas-api-xks-ca\") pod \"d131007a-17b7-4e82-9e38-96b96aed4932\" (UID: \"d131007a-17b7-4e82-9e38-96b96aed4932\") " Oct 30 21:46:39.857448 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:39.857314 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d131007a-17b7-4e82-9e38-96b96aed4932-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d131007a-17b7-4e82-9e38-96b96aed4932-maas-api-openshift-ca\") pod \"d131007a-17b7-4e82-9e38-96b96aed4932\" (UID: \"d131007a-17b7-4e82-9e38-96b96aed4932\") " Oct 30 21:46:39.857448 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:39.857421 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d131007a-17b7-4e82-9e38-96b96aed4932-kube-api-access-458qz\" (UniqueName: \"kubernetes.io/projected/d131007a-17b7-4e82-9e38-96b96aed4932-kube-api-access-458qz\") pod \"d131007a-17b7-4e82-9e38-96b96aed4932\" (UID: \"d131007a-17b7-4e82-9e38-96b96aed4932\") " Oct 30 21:46:39.857765 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:39.857730 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d131007a-17b7-4e82-9e38-96b96aed4932-maas-api-openshift-ca" pod "d131007a-17b7-4e82-9e38-96b96aed4932" (UID: "d131007a-17b7-4e82-9e38-96b96aed4932"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:39.859601 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:39.859563 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d131007a-17b7-4e82-9e38-96b96aed4932-maas-api-xks-ca" pod "d131007a-17b7-4e82-9e38-96b96aed4932" (UID: "d131007a-17b7-4e82-9e38-96b96aed4932"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:39.859601 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:39.859571 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d131007a-17b7-4e82-9e38-96b96aed4932-kube-api-access-458qz" pod "d131007a-17b7-4e82-9e38-96b96aed4932" (UID: "d131007a-17b7-4e82-9e38-96b96aed4932"). InnerVolumeSpecName "kube-api-access-458qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:39.958760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:39.958719 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d131007a-17b7-4e82-9e38-96b96aed4932-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.958760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:39.958756 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-458qz\" (UniqueName: \"kubernetes.io/projected/d131007a-17b7-4e82-9e38-96b96aed4932-kube-api-access-458qz\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:46:39.958760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:39.958769 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d131007a-17b7-4e82-9e38-96b96aed4932-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:46:40.589920 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:40.589887 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91171a3c-v5hdk" Oct 30 21:46:40.590141 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:40.589915 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91171a3c-v5hdk" event={"ID":"d131007a-17b7-4e82-9e38-96b96aed4932","Type":"ContainerDied","Data":"f7ca79f4a5f0fb4ac8b82966223224dd87219c7b06e88c870c9fd00c8c3552eb"} Oct 30 21:46:40.590141 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:40.589974 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7ca79f4a5f0fb4ac8b82966223224dd87219c7b06e88c870c9fd00c8c3552eb" Oct 30 21:46:41.017725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:41.017690 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e6b9efd-fc70-4387-9608-0a5dbbcc0a56" path="/var/lib/kubelet/pods/3e6b9efd-fc70-4387-9608-0a5dbbcc0a56/volumes" Oct 30 21:46:41.941055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:41.935664 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:46:46.855225 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:46.855167 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc"] Oct 30 21:46:46.857382 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:46.857315 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d131007a-17b7-4e82-9e38-96b96aed4932" containerName="revoke-keys" Oct 30 21:46:46.859127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:46.859101 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d131007a-17b7-4e82-9e38-96b96aed4932" containerName="revoke-keys" Oct 30 21:46:46.859793 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:46.859758 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d131007a-17b7-4e82-9e38-96b96aed4932" containerName="revoke-keys" Oct 30 21:46:46.869213 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:46.869185 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc" Oct 30 21:46:46.872944 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:46.872912 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc"] Oct 30 21:46:46.951737 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:46.951700 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:46:46.952096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:46.952075 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:46:46.961631 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:46.961597 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbxfd\" (UniqueName: \"kubernetes.io/projected/5f2d8855-6268-45d1-ba54-b27bb7913608-kube-api-access-cbxfd\") pod \"maas-api-revoke-sub-46946180-xfqtc\" (UID: \"5f2d8855-6268-45d1-ba54-b27bb7913608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc" Oct 30 21:46:46.962023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:46.961997 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2d8855-6268-45d1-ba54-b27bb7913608-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46946180-xfqtc\" (UID: \"5f2d8855-6268-45d1-ba54-b27bb7913608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc" Oct 30 21:46:46.962627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:46.962601 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2d8855-6268-45d1-ba54-b27bb7913608-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46946180-xfqtc\" (UID: \"5f2d8855-6268-45d1-ba54-b27bb7913608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc" Oct 30 21:46:47.097082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:47.081798 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2d8855-6268-45d1-ba54-b27bb7913608-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46946180-xfqtc\" (UID: \"5f2d8855-6268-45d1-ba54-b27bb7913608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc" Oct 30 21:46:47.097082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:47.081981 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbxfd\" (UniqueName: \"kubernetes.io/projected/5f2d8855-6268-45d1-ba54-b27bb7913608-kube-api-access-cbxfd\") pod \"maas-api-revoke-sub-46946180-xfqtc\" (UID: \"5f2d8855-6268-45d1-ba54-b27bb7913608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc" Oct 30 21:46:47.097082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:47.082181 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2d8855-6268-45d1-ba54-b27bb7913608-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46946180-xfqtc\" (UID: \"5f2d8855-6268-45d1-ba54-b27bb7913608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc" Oct 30 21:46:47.097082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:47.083364 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2d8855-6268-45d1-ba54-b27bb7913608-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46946180-xfqtc\" (UID: \"5f2d8855-6268-45d1-ba54-b27bb7913608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc" Oct 30 21:46:47.116146 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:47.116067 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbxfd\" (UniqueName: \"kubernetes.io/projected/5f2d8855-6268-45d1-ba54-b27bb7913608-kube-api-access-cbxfd\") pod \"maas-api-revoke-sub-46946180-xfqtc\" (UID: \"5f2d8855-6268-45d1-ba54-b27bb7913608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc" Oct 30 21:46:47.118065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:47.116569 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2d8855-6268-45d1-ba54-b27bb7913608-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46946180-xfqtc\" (UID: \"5f2d8855-6268-45d1-ba54-b27bb7913608\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc" Oct 30 21:46:47.303068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:47.302270 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc" Oct 30 21:46:47.309129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:47.305675 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr"] Oct 30 21:46:47.349064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:47.348390 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr" Oct 30 21:46:47.349746 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:47.348279 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr"] Oct 30 21:46:47.420130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:47.410937 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f77ba38-3140-4f05-b127-ea27b1967882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-276c4241-xsscr\" (UID: \"9f77ba38-3140-4f05-b127-ea27b1967882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr" Oct 30 21:46:47.420130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:47.411009 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f77ba38-3140-4f05-b127-ea27b1967882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-276c4241-xsscr\" (UID: \"9f77ba38-3140-4f05-b127-ea27b1967882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr" Oct 30 21:46:47.420130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:47.411077 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmwvd\" (UniqueName: \"kubernetes.io/projected/9f77ba38-3140-4f05-b127-ea27b1967882-kube-api-access-fmwvd\") pod \"maas-api-revoke-sub-276c4241-xsscr\" (UID: \"9f77ba38-3140-4f05-b127-ea27b1967882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr" Oct 30 21:46:47.540799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:47.540731 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f77ba38-3140-4f05-b127-ea27b1967882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-276c4241-xsscr\" (UID: \"9f77ba38-3140-4f05-b127-ea27b1967882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr" Oct 30 21:46:47.542067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:47.542020 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f77ba38-3140-4f05-b127-ea27b1967882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-276c4241-xsscr\" (UID: \"9f77ba38-3140-4f05-b127-ea27b1967882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr" Oct 30 21:46:47.567058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:47.558891 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmwvd\" (UniqueName: \"kubernetes.io/projected/9f77ba38-3140-4f05-b127-ea27b1967882-kube-api-access-fmwvd\") pod \"maas-api-revoke-sub-276c4241-xsscr\" (UID: \"9f77ba38-3140-4f05-b127-ea27b1967882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr" Oct 30 21:46:47.567058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:47.541911 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f77ba38-3140-4f05-b127-ea27b1967882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-276c4241-xsscr\" (UID: \"9f77ba38-3140-4f05-b127-ea27b1967882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr" Oct 30 21:46:47.638076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:47.630917 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f77ba38-3140-4f05-b127-ea27b1967882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-276c4241-xsscr\" (UID: \"9f77ba38-3140-4f05-b127-ea27b1967882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr" Oct 30 21:46:47.639109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:47.633800 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmwvd\" (UniqueName: \"kubernetes.io/projected/9f77ba38-3140-4f05-b127-ea27b1967882-kube-api-access-fmwvd\") pod \"maas-api-revoke-sub-276c4241-xsscr\" (UID: \"9f77ba38-3140-4f05-b127-ea27b1967882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr" Oct 30 21:46:47.659061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:47.654848 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr" Oct 30 21:46:48.281772 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:48.280418 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc"] Oct 30 21:46:48.660186 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:48.658825 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr"] Oct 30 21:46:48.663956 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:46:48.663922 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f77ba38_3140_4f05_b127_ea27b1967882.slice/crio-34190290a425c418bc40e4d66e0935ac3c53259f9dbf6c76caa91149d293a71d WatchSource:0}: Error finding container 34190290a425c418bc40e4d66e0935ac3c53259f9dbf6c76caa91149d293a71d: Status 404 returned error can't find the container with id 34190290a425c418bc40e4d66e0935ac3c53259f9dbf6c76caa91149d293a71d Oct 30 21:46:49.262622 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:49.262531 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr" event={"ID":"9f77ba38-3140-4f05-b127-ea27b1967882","Type":"ContainerStarted","Data":"f037d1f911c9d69ecd76c32e1029cdebc53e30159813bda6c05719dbbf02d958"} Oct 30 21:46:49.262622 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:49.262598 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr" event={"ID":"9f77ba38-3140-4f05-b127-ea27b1967882","Type":"ContainerStarted","Data":"34190290a425c418bc40e4d66e0935ac3c53259f9dbf6c76caa91149d293a71d"} Oct 30 21:46:49.264406 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:49.264378 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc" event={"ID":"5f2d8855-6268-45d1-ba54-b27bb7913608","Type":"ContainerStarted","Data":"3a6002f00dac67bb409b44fb30a15a09de17474ed59f850de00aedc964178f74"} Oct 30 21:46:49.264533 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:49.264433 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc" event={"ID":"5f2d8855-6268-45d1-ba54-b27bb7913608","Type":"ContainerStarted","Data":"afb63322c717e653d7c749c7e8efb461963476fdef0808076b74b4c2e084828b"} Oct 30 21:46:49.332430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:49.332360 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr" podStartSLOduration=2.332338747 podStartE2EDuration="2.332338747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:49.324465865 +0000 UTC m=+2796.445539615" watchObservedRunningTime="2026-10-30 21:46:49.332338747 +0000 UTC m=+2796.453412498" Oct 30 21:46:49.396753 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:49.388708 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc" podStartSLOduration=3.388674306 podStartE2EDuration="3.388674306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:49.36345517 +0000 UTC m=+2796.484528918" watchObservedRunningTime="2026-10-30 21:46:49.388674306 +0000 UTC m=+2796.509748054" Oct 30 21:46:50.278568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:50.278313 4926 generic.go:356] "Generic (PLEG): container finished" podID="9f77ba38-3140-4f05-b127-ea27b1967882" containerID="f037d1f911c9d69ecd76c32e1029cdebc53e30159813bda6c05719dbbf02d958" exitCode=0 Oct 30 21:46:50.278568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:50.278358 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr" event={"ID":"9f77ba38-3140-4f05-b127-ea27b1967882","Type":"ContainerDied","Data":"f037d1f911c9d69ecd76c32e1029cdebc53e30159813bda6c05719dbbf02d958"} Oct 30 21:46:50.280924 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:50.280895 4926 generic.go:356] "Generic (PLEG): container finished" podID="5f2d8855-6268-45d1-ba54-b27bb7913608" containerID="3a6002f00dac67bb409b44fb30a15a09de17474ed59f850de00aedc964178f74" exitCode=0 Oct 30 21:46:50.281274 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:50.280972 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc" event={"ID":"5f2d8855-6268-45d1-ba54-b27bb7913608","Type":"ContainerDied","Data":"3a6002f00dac67bb409b44fb30a15a09de17474ed59f850de00aedc964178f74"} Oct 30 21:46:51.539934 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:51.539896 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr" Oct 30 21:46:51.616091 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:51.615451 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc" Oct 30 21:46:51.619388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:51.619357 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f77ba38-3140-4f05-b127-ea27b1967882-kube-api-access-fmwvd\" (UniqueName: \"kubernetes.io/projected/9f77ba38-3140-4f05-b127-ea27b1967882-kube-api-access-fmwvd\") pod \"9f77ba38-3140-4f05-b127-ea27b1967882\" (UID: \"9f77ba38-3140-4f05-b127-ea27b1967882\") " Oct 30 21:46:51.619568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:51.619426 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f77ba38-3140-4f05-b127-ea27b1967882-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f77ba38-3140-4f05-b127-ea27b1967882-maas-api-openshift-ca\") pod \"9f77ba38-3140-4f05-b127-ea27b1967882\" (UID: \"9f77ba38-3140-4f05-b127-ea27b1967882\") " Oct 30 21:46:51.619568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:51.619523 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f77ba38-3140-4f05-b127-ea27b1967882-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f77ba38-3140-4f05-b127-ea27b1967882-maas-api-xks-ca\") pod \"9f77ba38-3140-4f05-b127-ea27b1967882\" (UID: \"9f77ba38-3140-4f05-b127-ea27b1967882\") " Oct 30 21:46:51.620929 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:51.620896 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f77ba38-3140-4f05-b127-ea27b1967882-maas-api-openshift-ca" pod "9f77ba38-3140-4f05-b127-ea27b1967882" (UID: "9f77ba38-3140-4f05-b127-ea27b1967882"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:51.626269 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:51.626135 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f77ba38-3140-4f05-b127-ea27b1967882-maas-api-xks-ca" pod "9f77ba38-3140-4f05-b127-ea27b1967882" (UID: "9f77ba38-3140-4f05-b127-ea27b1967882"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:51.626269 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:51.626238 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f77ba38-3140-4f05-b127-ea27b1967882-kube-api-access-fmwvd" pod "9f77ba38-3140-4f05-b127-ea27b1967882" (UID: "9f77ba38-3140-4f05-b127-ea27b1967882"). InnerVolumeSpecName "kube-api-access-fmwvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:51.720979 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:51.720929 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2d8855-6268-45d1-ba54-b27bb7913608-kube-api-access-cbxfd\" (UniqueName: \"kubernetes.io/projected/5f2d8855-6268-45d1-ba54-b27bb7913608-kube-api-access-cbxfd\") pod \"5f2d8855-6268-45d1-ba54-b27bb7913608\" (UID: \"5f2d8855-6268-45d1-ba54-b27bb7913608\") " Oct 30 21:46:51.721238 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:51.721000 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f2d8855-6268-45d1-ba54-b27bb7913608-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2d8855-6268-45d1-ba54-b27bb7913608-maas-api-xks-ca\") pod \"5f2d8855-6268-45d1-ba54-b27bb7913608\" (UID: \"5f2d8855-6268-45d1-ba54-b27bb7913608\") " Oct 30 21:46:51.721238 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:51.721150 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f2d8855-6268-45d1-ba54-b27bb7913608-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2d8855-6268-45d1-ba54-b27bb7913608-maas-api-openshift-ca\") pod \"5f2d8855-6268-45d1-ba54-b27bb7913608\" (UID: \"5f2d8855-6268-45d1-ba54-b27bb7913608\") " Oct 30 21:46:51.722365 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:51.722336 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2d8855-6268-45d1-ba54-b27bb7913608-maas-api-openshift-ca" pod "5f2d8855-6268-45d1-ba54-b27bb7913608" (UID: "5f2d8855-6268-45d1-ba54-b27bb7913608"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:46:51.722658 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:51.722635 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f77ba38-3140-4f05-b127-ea27b1967882-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:46:51.722792 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:51.722777 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f77ba38-3140-4f05-b127-ea27b1967882-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:46:51.722902 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:51.722887 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmwvd\" (UniqueName: \"kubernetes.io/projected/9f77ba38-3140-4f05-b127-ea27b1967882-kube-api-access-fmwvd\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:46:51.724589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:51.724547 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f2d8855-6268-45d1-ba54-b27bb7913608-maas-api-xks-ca" pod "5f2d8855-6268-45d1-ba54-b27bb7913608" (UID: "5f2d8855-6268-45d1-ba54-b27bb7913608"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:51.724958 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:51.724937 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2d8855-6268-45d1-ba54-b27bb7913608-kube-api-access-cbxfd" pod "5f2d8855-6268-45d1-ba54-b27bb7913608" (UID: "5f2d8855-6268-45d1-ba54-b27bb7913608"). InnerVolumeSpecName "kube-api-access-cbxfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:51.825285 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:51.825201 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbxfd\" (UniqueName: \"kubernetes.io/projected/5f2d8855-6268-45d1-ba54-b27bb7913608-kube-api-access-cbxfd\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:46:51.825285 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:51.825241 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f2d8855-6268-45d1-ba54-b27bb7913608-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:46:51.825285 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:51.825256 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f2d8855-6268-45d1-ba54-b27bb7913608-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:46:52.311408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:52.311359 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr" event={"ID":"9f77ba38-3140-4f05-b127-ea27b1967882","Type":"ContainerDied","Data":"34190290a425c418bc40e4d66e0935ac3c53259f9dbf6c76caa91149d293a71d"} Oct 30 21:46:52.311629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:52.311436 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34190290a425c418bc40e4d66e0935ac3c53259f9dbf6c76caa91149d293a71d" Oct 30 21:46:52.311629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:52.311520 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr" Oct 30 21:46:52.318016 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:52.317976 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc" event={"ID":"5f2d8855-6268-45d1-ba54-b27bb7913608","Type":"ContainerDied","Data":"afb63322c717e653d7c749c7e8efb461963476fdef0808076b74b4c2e084828b"} Oct 30 21:46:52.318202 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:52.318086 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afb63322c717e653d7c749c7e8efb461963476fdef0808076b74b4c2e084828b" Oct 30 21:46:52.318202 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:52.318169 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc" Oct 30 21:46:54.338637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:54.338548 4926 generic.go:356] "Generic (PLEG): container finished" podID="50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" containerID="2043dd192beaa7e842e844f63a0681d3dee2910e3ca5150a364a7e4f4a2853fb" exitCode=0 Oct 30 21:46:54.338637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:54.338620 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" event={"ID":"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5","Type":"ContainerDied","Data":"2043dd192beaa7e842e844f63a0681d3dee2910e3ca5150a364a7e4f4a2853fb"} Oct 30 21:46:54.339184 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:54.339171 4926 scope.go:122] "RemoveContainer" containerID="2043dd192beaa7e842e844f63a0681d3dee2910e3ca5150a364a7e4f4a2853fb" Oct 30 21:46:54.340753 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:54.340732 4926 generic.go:356] "Generic (PLEG): container finished" podID="c49c7689-1388-4442-8ea8-825da60fab98" containerID="2b102897365236e348c1ba81bd40f11f89da75a791b377b6132bb10e0554394d" exitCode=0 Oct 30 21:46:54.340851 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:54.340795 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" event={"ID":"c49c7689-1388-4442-8ea8-825da60fab98","Type":"ContainerDied","Data":"2b102897365236e348c1ba81bd40f11f89da75a791b377b6132bb10e0554394d"} Oct 30 21:46:54.341227 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:54.341209 4926 scope.go:122] "RemoveContainer" containerID="2b102897365236e348c1ba81bd40f11f89da75a791b377b6132bb10e0554394d" Oct 30 21:46:54.341294 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:54.341241 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:54.341351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:54.341325 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:54.341406 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:54.341356 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:54.342952 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:54.342921 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:54.343044 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:54.342991 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:54.343110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:54.343077 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:46:55.431056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:55.430847 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" event={"ID":"c49c7689-1388-4442-8ea8-825da60fab98","Type":"ContainerStarted","Data":"ff397e6e0c9ebff2b2a24402f6d517d7744d7ab873981ac1c93dc040e6706fc2"} Oct 30 21:46:55.451108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:55.447867 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" event={"ID":"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5","Type":"ContainerStarted","Data":"39b11732a64b796d59ccc618e70c6e4e1f4c6cad62f1ca620736ae58ec607468"} Oct 30 21:46:55.702640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:55.702533 4926 patch_prober.go:29] interesting pod/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 21:46:55.702962 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:55.702913 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" podUID="50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 21:46:55.703153 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:55.703081 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:46:55.850362 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:55.850270 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:46:56.204127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:56.204092 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:46:56.206398 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:56.206365 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 21:46:56.206655 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:56.206628 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" podUID="c49c7689-1388-4442-8ea8-825da60fab98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 21:46:56.706055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:56.705452 4926 patch_prober.go:29] interesting pod/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 21:46:56.706055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:56.705534 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" podUID="50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 21:46:57.016110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:56.997103 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb"] Oct 30 21:46:57.016110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:56.998111 4926 cpu_manager.go:414] "containerMap: removing container" podUID="5f2d8855-6268-45d1-ba54-b27bb7913608" containerName="revoke-keys" Oct 30 21:46:57.016110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:56.998137 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2d8855-6268-45d1-ba54-b27bb7913608" containerName="revoke-keys" Oct 30 21:46:57.016110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:56.998153 4926 cpu_manager.go:414] "containerMap: removing container" podUID="9f77ba38-3140-4f05-b127-ea27b1967882" containerName="revoke-keys" Oct 30 21:46:57.016110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:56.998162 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f77ba38-3140-4f05-b127-ea27b1967882" containerName="revoke-keys" Oct 30 21:46:57.016110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:56.998363 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f2d8855-6268-45d1-ba54-b27bb7913608" containerName="revoke-keys" Oct 30 21:46:57.016110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:56.998376 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f77ba38-3140-4f05-b127-ea27b1967882" containerName="revoke-keys" Oct 30 21:46:57.041059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:57.037303 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb" Oct 30 21:46:57.057928 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:57.054734 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219457b4-60ac-4501-aeda-f40528bddee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1068fa75-rmqfb\" (UID: \"219457b4-60ac-4501-aeda-f40528bddee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb" Oct 30 21:46:57.057928 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:57.054940 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fb8w\" (UniqueName: \"kubernetes.io/projected/219457b4-60ac-4501-aeda-f40528bddee1-kube-api-access-2fb8w\") pod \"maas-api-revoke-sub-1068fa75-rmqfb\" (UID: \"219457b4-60ac-4501-aeda-f40528bddee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb" Oct 30 21:46:57.057928 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:57.055066 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219457b4-60ac-4501-aeda-f40528bddee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1068fa75-rmqfb\" (UID: \"219457b4-60ac-4501-aeda-f40528bddee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb" Oct 30 21:46:57.057928 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:57.057503 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:46:57.058915 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:57.058302 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:46:57.060426 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:57.060402 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb"] Oct 30 21:46:57.156791 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:57.156743 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219457b4-60ac-4501-aeda-f40528bddee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1068fa75-rmqfb\" (UID: \"219457b4-60ac-4501-aeda-f40528bddee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb" Oct 30 21:46:57.157083 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:57.156966 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fb8w\" (UniqueName: \"kubernetes.io/projected/219457b4-60ac-4501-aeda-f40528bddee1-kube-api-access-2fb8w\") pod \"maas-api-revoke-sub-1068fa75-rmqfb\" (UID: \"219457b4-60ac-4501-aeda-f40528bddee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb" Oct 30 21:46:57.157178 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:57.157082 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219457b4-60ac-4501-aeda-f40528bddee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1068fa75-rmqfb\" (UID: \"219457b4-60ac-4501-aeda-f40528bddee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb" Oct 30 21:46:57.159064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:57.159016 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219457b4-60ac-4501-aeda-f40528bddee1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1068fa75-rmqfb\" (UID: \"219457b4-60ac-4501-aeda-f40528bddee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb" Oct 30 21:46:57.160558 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:57.160533 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219457b4-60ac-4501-aeda-f40528bddee1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1068fa75-rmqfb\" (UID: \"219457b4-60ac-4501-aeda-f40528bddee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb" Oct 30 21:46:57.241476 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:57.241436 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 21:46:57.241778 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:57.241745 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" podUID="c49c7689-1388-4442-8ea8-825da60fab98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 21:46:57.243100 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:57.243071 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fb8w\" (UniqueName: \"kubernetes.io/projected/219457b4-60ac-4501-aeda-f40528bddee1-kube-api-access-2fb8w\") pod \"maas-api-revoke-sub-1068fa75-rmqfb\" (UID: \"219457b4-60ac-4501-aeda-f40528bddee1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb" Oct 30 21:46:57.363463 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:57.363177 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb" Oct 30 21:46:57.797058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:57.781617 4926 patch_prober.go:29] interesting pod/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 21:46:57.810203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:57.810151 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" podUID="50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 21:46:58.225133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:58.209483 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 21:46:58.225133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:58.209570 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" podUID="c49c7689-1388-4442-8ea8-825da60fab98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 21:46:58.320657 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:58.287660 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb"] Oct 30 21:46:58.350102 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:46:58.346308 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod219457b4_60ac_4501_aeda_f40528bddee1.slice/crio-ee84c62fe99acb590758185f379189ceba9a3a3110bdd24bac730da19f25abb4 WatchSource:0}: Error finding container ee84c62fe99acb590758185f379189ceba9a3a3110bdd24bac730da19f25abb4: Status 404 returned error can't find the container with id ee84c62fe99acb590758185f379189ceba9a3a3110bdd24bac730da19f25abb4 Oct 30 21:46:58.798058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:58.796783 4926 patch_prober.go:29] interesting pod/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 21:46:58.798058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:58.796870 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" podUID="50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 21:46:58.852151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:58.852080 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb" event={"ID":"219457b4-60ac-4501-aeda-f40528bddee1","Type":"ContainerStarted","Data":"ee84c62fe99acb590758185f379189ceba9a3a3110bdd24bac730da19f25abb4"} Oct 30 21:46:59.204060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:59.203392 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 21:46:59.204060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:59.203471 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" podUID="c49c7689-1388-4442-8ea8-825da60fab98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 21:46:59.706057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:59.701540 4926 patch_prober.go:29] interesting pod/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 21:46:59.706057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:59.701617 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" podUID="50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 21:46:59.865227 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:59.865177 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb" event={"ID":"219457b4-60ac-4501-aeda-f40528bddee1","Type":"ContainerStarted","Data":"4f44368f926c180adf26a507562d4a8643ecd0fc2fc63b972d9e10c746968321"} Oct 30 21:46:59.923089 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:46:59.922999 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb" podStartSLOduration=3.922976049 podStartE2EDuration="3.922976049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:59.91377242 +0000 UTC m=+2807.034846167" watchObservedRunningTime="2026-10-30 21:46:59.922976049 +0000 UTC m=+2807.044049797" Oct 30 21:47:00.203059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:00.202409 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 21:47:00.203059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:00.202487 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" podUID="c49c7689-1388-4442-8ea8-825da60fab98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 21:47:00.702867 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:00.702412 4926 patch_prober.go:29] interesting pod/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 30 21:47:00.702867 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:00.702478 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" podUID="50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 30 21:47:00.915767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:00.915728 4926 generic.go:356] "Generic (PLEG): container finished" podID="219457b4-60ac-4501-aeda-f40528bddee1" containerID="4f44368f926c180adf26a507562d4a8643ecd0fc2fc63b972d9e10c746968321" exitCode=0 Oct 30 21:47:00.920942 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:00.918520 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb" event={"ID":"219457b4-60ac-4501-aeda-f40528bddee1","Type":"ContainerDied","Data":"4f44368f926c180adf26a507562d4a8643ecd0fc2fc63b972d9e10c746968321"} Oct 30 21:47:01.221056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:01.219585 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" start-of-body= Oct 30 21:47:01.221056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:01.219678 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" podUID="c49c7689-1388-4442-8ea8-825da60fab98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.227:15021/healthz/ready\": dial tcp 10.132.0.227:15021: connect: connection refused" Oct 30 21:47:01.868767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:01.868704 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:47:01.870523 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:01.870495 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:47:01.880877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:01.880830 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:47:02.129349 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:02.129277 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb" Oct 30 21:47:02.203891 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:02.203852 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/219457b4-60ac-4501-aeda-f40528bddee1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219457b4-60ac-4501-aeda-f40528bddee1-maas-api-openshift-ca\") pod \"219457b4-60ac-4501-aeda-f40528bddee1\" (UID: \"219457b4-60ac-4501-aeda-f40528bddee1\") " Oct 30 21:47:02.204598 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:02.204565 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/219457b4-60ac-4501-aeda-f40528bddee1-maas-api-openshift-ca" pod "219457b4-60ac-4501-aeda-f40528bddee1" (UID: "219457b4-60ac-4501-aeda-f40528bddee1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:02.204902 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:02.204877 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219457b4-60ac-4501-aeda-f40528bddee1-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:47:02.215474 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:02.215442 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:47:02.218204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:02.218179 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:47:02.306012 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:02.305967 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/219457b4-60ac-4501-aeda-f40528bddee1-kube-api-access-2fb8w\" (UniqueName: \"kubernetes.io/projected/219457b4-60ac-4501-aeda-f40528bddee1-kube-api-access-2fb8w\") pod \"219457b4-60ac-4501-aeda-f40528bddee1\" (UID: \"219457b4-60ac-4501-aeda-f40528bddee1\") " Oct 30 21:47:02.306281 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:02.306128 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/219457b4-60ac-4501-aeda-f40528bddee1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219457b4-60ac-4501-aeda-f40528bddee1-maas-api-xks-ca\") pod \"219457b4-60ac-4501-aeda-f40528bddee1\" (UID: \"219457b4-60ac-4501-aeda-f40528bddee1\") " Oct 30 21:47:02.308923 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:02.308888 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/219457b4-60ac-4501-aeda-f40528bddee1-kube-api-access-2fb8w" pod "219457b4-60ac-4501-aeda-f40528bddee1" (UID: "219457b4-60ac-4501-aeda-f40528bddee1"). InnerVolumeSpecName "kube-api-access-2fb8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:02.310197 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:02.310165 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/219457b4-60ac-4501-aeda-f40528bddee1-maas-api-xks-ca" pod "219457b4-60ac-4501-aeda-f40528bddee1" (UID: "219457b4-60ac-4501-aeda-f40528bddee1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:02.494172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:02.493931 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fb8w\" (UniqueName: \"kubernetes.io/projected/219457b4-60ac-4501-aeda-f40528bddee1-kube-api-access-2fb8w\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:47:02.501979 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:02.501936 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219457b4-60ac-4501-aeda-f40528bddee1-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:47:02.937776 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:02.937744 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb" Oct 30 21:47:02.937968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:02.937779 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb" event={"ID":"219457b4-60ac-4501-aeda-f40528bddee1","Type":"ContainerDied","Data":"ee84c62fe99acb590758185f379189ceba9a3a3110bdd24bac730da19f25abb4"} Oct 30 21:47:02.937968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:02.937826 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee84c62fe99acb590758185f379189ceba9a3a3110bdd24bac730da19f25abb4" Oct 30 21:47:07.487390 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:47:07.487344 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:47:16.645761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:16.642897 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9"] Oct 30 21:47:16.645761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:16.643839 4926 cpu_manager.go:414] "containerMap: removing container" podUID="219457b4-60ac-4501-aeda-f40528bddee1" containerName="revoke-keys" Oct 30 21:47:16.645761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:16.643864 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="219457b4-60ac-4501-aeda-f40528bddee1" containerName="revoke-keys" Oct 30 21:47:16.645761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:16.644094 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="219457b4-60ac-4501-aeda-f40528bddee1" containerName="revoke-keys" Oct 30 21:47:16.650894 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:16.650868 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9" Oct 30 21:47:16.676419 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:16.676370 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9"] Oct 30 21:47:16.678237 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:16.678202 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:47:16.689336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:16.689278 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:47:16.708567 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:16.708525 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acea6911-e45b-461a-b5b2-3ac1a5a637c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9\" (UID: \"acea6911-e45b-461a-b5b2-3ac1a5a637c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9" Oct 30 21:47:16.708795 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:16.708757 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjvv5\" (UniqueName: \"kubernetes.io/projected/acea6911-e45b-461a-b5b2-3ac1a5a637c7-kube-api-access-pjvv5\") pod \"maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9\" (UID: \"acea6911-e45b-461a-b5b2-3ac1a5a637c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9" Oct 30 21:47:16.708795 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:16.708786 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acea6911-e45b-461a-b5b2-3ac1a5a637c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9\" (UID: \"acea6911-e45b-461a-b5b2-3ac1a5a637c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9" Oct 30 21:47:16.810110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:16.810069 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acea6911-e45b-461a-b5b2-3ac1a5a637c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9\" (UID: \"acea6911-e45b-461a-b5b2-3ac1a5a637c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9" Oct 30 21:47:16.810339 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:16.810294 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjvv5\" (UniqueName: \"kubernetes.io/projected/acea6911-e45b-461a-b5b2-3ac1a5a637c7-kube-api-access-pjvv5\") pod \"maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9\" (UID: \"acea6911-e45b-461a-b5b2-3ac1a5a637c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9" Oct 30 21:47:16.810339 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:16.810328 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acea6911-e45b-461a-b5b2-3ac1a5a637c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9\" (UID: \"acea6911-e45b-461a-b5b2-3ac1a5a637c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9" Oct 30 21:47:16.810902 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:16.810877 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acea6911-e45b-461a-b5b2-3ac1a5a637c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9\" (UID: \"acea6911-e45b-461a-b5b2-3ac1a5a637c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9" Oct 30 21:47:16.813078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:16.813019 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acea6911-e45b-461a-b5b2-3ac1a5a637c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9\" (UID: \"acea6911-e45b-461a-b5b2-3ac1a5a637c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9" Oct 30 21:47:16.828927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:16.828887 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjvv5\" (UniqueName: \"kubernetes.io/projected/acea6911-e45b-461a-b5b2-3ac1a5a637c7-kube-api-access-pjvv5\") pod \"maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9\" (UID: \"acea6911-e45b-461a-b5b2-3ac1a5a637c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9" Oct 30 21:47:16.996851 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:16.996565 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9" Oct 30 21:47:17.324024 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:17.320885 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9"] Oct 30 21:47:18.086381 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:18.086297 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9" event={"ID":"acea6911-e45b-461a-b5b2-3ac1a5a637c7","Type":"ContainerStarted","Data":"127ff05bb830c630034cc7270e3e202b184802db0e8f5417efadb66e13ba1694"} Oct 30 21:47:18.086381 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:18.086356 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9" event={"ID":"acea6911-e45b-461a-b5b2-3ac1a5a637c7","Type":"ContainerStarted","Data":"39a93b25ee10cfd625f9ce6c1fb3f30285c2aa166836c924adfb357fa3fd8b45"} Oct 30 21:47:18.128144 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:18.128079 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9" podStartSLOduration=2.128060182 podStartE2EDuration="2.128060182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:18.123508807 +0000 UTC m=+2825.244582555" watchObservedRunningTime="2026-10-30 21:47:18.128060182 +0000 UTC m=+2825.249133930" Oct 30 21:47:19.098254 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:19.098214 4926 generic.go:356] "Generic (PLEG): container finished" podID="acea6911-e45b-461a-b5b2-3ac1a5a637c7" containerID="127ff05bb830c630034cc7270e3e202b184802db0e8f5417efadb66e13ba1694" exitCode=0 Oct 30 21:47:19.098679 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:19.098298 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9" event={"ID":"acea6911-e45b-461a-b5b2-3ac1a5a637c7","Type":"ContainerDied","Data":"127ff05bb830c630034cc7270e3e202b184802db0e8f5417efadb66e13ba1694"} Oct 30 21:47:20.305128 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:20.305095 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9" Oct 30 21:47:20.462841 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:20.462801 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acea6911-e45b-461a-b5b2-3ac1a5a637c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acea6911-e45b-461a-b5b2-3ac1a5a637c7-maas-api-openshift-ca\") pod \"acea6911-e45b-461a-b5b2-3ac1a5a637c7\" (UID: \"acea6911-e45b-461a-b5b2-3ac1a5a637c7\") " Oct 30 21:47:20.463092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:20.462997 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acea6911-e45b-461a-b5b2-3ac1a5a637c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acea6911-e45b-461a-b5b2-3ac1a5a637c7-maas-api-xks-ca\") pod \"acea6911-e45b-461a-b5b2-3ac1a5a637c7\" (UID: \"acea6911-e45b-461a-b5b2-3ac1a5a637c7\") " Oct 30 21:47:20.463092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:20.463027 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acea6911-e45b-461a-b5b2-3ac1a5a637c7-kube-api-access-pjvv5\" (UniqueName: \"kubernetes.io/projected/acea6911-e45b-461a-b5b2-3ac1a5a637c7-kube-api-access-pjvv5\") pod \"acea6911-e45b-461a-b5b2-3ac1a5a637c7\" (UID: \"acea6911-e45b-461a-b5b2-3ac1a5a637c7\") " Oct 30 21:47:20.463267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:20.463239 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acea6911-e45b-461a-b5b2-3ac1a5a637c7-maas-api-openshift-ca" pod "acea6911-e45b-461a-b5b2-3ac1a5a637c7" (UID: "acea6911-e45b-461a-b5b2-3ac1a5a637c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:20.463719 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:20.463694 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acea6911-e45b-461a-b5b2-3ac1a5a637c7-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:47:20.465441 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:20.465414 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acea6911-e45b-461a-b5b2-3ac1a5a637c7-maas-api-xks-ca" pod "acea6911-e45b-461a-b5b2-3ac1a5a637c7" (UID: "acea6911-e45b-461a-b5b2-3ac1a5a637c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:20.465586 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:20.465562 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acea6911-e45b-461a-b5b2-3ac1a5a637c7-kube-api-access-pjvv5" pod "acea6911-e45b-461a-b5b2-3ac1a5a637c7" (UID: "acea6911-e45b-461a-b5b2-3ac1a5a637c7"). InnerVolumeSpecName "kube-api-access-pjvv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:20.564808 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:20.564767 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acea6911-e45b-461a-b5b2-3ac1a5a637c7-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:47:20.564808 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:20.564801 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjvv5\" (UniqueName: \"kubernetes.io/projected/acea6911-e45b-461a-b5b2-3ac1a5a637c7-kube-api-access-pjvv5\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:47:21.115842 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:21.115811 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9" Oct 30 21:47:21.116025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:21.115814 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9" event={"ID":"acea6911-e45b-461a-b5b2-3ac1a5a637c7","Type":"ContainerDied","Data":"39a93b25ee10cfd625f9ce6c1fb3f30285c2aa166836c924adfb357fa3fd8b45"} Oct 30 21:47:21.116025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:21.115935 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39a93b25ee10cfd625f9ce6c1fb3f30285c2aa166836c924adfb357fa3fd8b45" Oct 30 21:47:21.543711 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:21.543668 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5"] Oct 30 21:47:21.544288 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:21.543866 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" podUID="bb16f268-1822-483a-bdca-87bafd95625e" containerName="istio-proxy" containerID="cri-o://1ac366253e8b8b9f2a35e4ac19df45333893f878710b6218673569bfbec59de0" gracePeriod=30 Oct 30 21:47:26.849443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.849412 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:47:26.962348 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.962312 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-workload-certs\") pod \"bb16f268-1822-483a-bdca-87bafd95625e\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " Oct 30 21:47:26.962583 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.962412 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-workload-socket\") pod \"bb16f268-1822-483a-bdca-87bafd95625e\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " Oct 30 21:47:26.962583 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.962482 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-istio-envoy\") pod \"bb16f268-1822-483a-bdca-87bafd95625e\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " Oct 30 21:47:26.962583 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.962515 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-istio-data\") pod \"bb16f268-1822-483a-bdca-87bafd95625e\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " Oct 30 21:47:26.962722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.962612 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb16f268-1822-483a-bdca-87bafd95625e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb16f268-1822-483a-bdca-87bafd95625e-istiod-ca-cert\") pod \"bb16f268-1822-483a-bdca-87bafd95625e\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " Oct 30 21:47:26.962722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.962629 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-workload-certs" pod "bb16f268-1822-483a-bdca-87bafd95625e" (UID: "bb16f268-1822-483a-bdca-87bafd95625e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:26.962722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.962650 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb16f268-1822-483a-bdca-87bafd95625e-kube-api-access-bsmmz\" (UniqueName: \"kubernetes.io/projected/bb16f268-1822-483a-bdca-87bafd95625e-kube-api-access-bsmmz\") pod \"bb16f268-1822-483a-bdca-87bafd95625e\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " Oct 30 21:47:26.962899 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.962721 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-workload-socket" pod "bb16f268-1822-483a-bdca-87bafd95625e" (UID: "bb16f268-1822-483a-bdca-87bafd95625e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:26.962899 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.962779 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb16f268-1822-483a-bdca-87bafd95625e-istio-token\" (UniqueName: \"kubernetes.io/projected/bb16f268-1822-483a-bdca-87bafd95625e-istio-token\") pod \"bb16f268-1822-483a-bdca-87bafd95625e\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " Oct 30 21:47:26.962899 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.962784 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-istio-data" pod "bb16f268-1822-483a-bdca-87bafd95625e" (UID: "bb16f268-1822-483a-bdca-87bafd95625e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:26.962899 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.962867 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-credential-socket\") pod \"bb16f268-1822-483a-bdca-87bafd95625e\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " Oct 30 21:47:26.963217 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.962936 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb16f268-1822-483a-bdca-87bafd95625e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb16f268-1822-483a-bdca-87bafd95625e-istio-podinfo\") pod \"bb16f268-1822-483a-bdca-87bafd95625e\" (UID: \"bb16f268-1822-483a-bdca-87bafd95625e\") " Oct 30 21:47:26.963217 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.963107 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-credential-socket" pod "bb16f268-1822-483a-bdca-87bafd95625e" (UID: "bb16f268-1822-483a-bdca-87bafd95625e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:26.963217 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.963150 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb16f268-1822-483a-bdca-87bafd95625e-istiod-ca-cert" pod "bb16f268-1822-483a-bdca-87bafd95625e" (UID: "bb16f268-1822-483a-bdca-87bafd95625e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:26.963595 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.963571 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:47:26.963703 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.963600 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:47:26.963703 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.963613 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb16f268-1822-483a-bdca-87bafd95625e-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:47:26.963703 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.963625 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:47:26.963703 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.963637 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:47:26.965018 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.964990 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-istio-envoy" pod "bb16f268-1822-483a-bdca-87bafd95625e" (UID: "bb16f268-1822-483a-bdca-87bafd95625e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:47:26.965231 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.965209 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb16f268-1822-483a-bdca-87bafd95625e-istio-podinfo" pod "bb16f268-1822-483a-bdca-87bafd95625e" (UID: "bb16f268-1822-483a-bdca-87bafd95625e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:47:26.965346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.965329 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb16f268-1822-483a-bdca-87bafd95625e-istio-token" pod "bb16f268-1822-483a-bdca-87bafd95625e" (UID: "bb16f268-1822-483a-bdca-87bafd95625e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:26.965389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:26.965352 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb16f268-1822-483a-bdca-87bafd95625e-kube-api-access-bsmmz" pod "bb16f268-1822-483a-bdca-87bafd95625e" (UID: "bb16f268-1822-483a-bdca-87bafd95625e"). InnerVolumeSpecName "kube-api-access-bsmmz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:27.065468 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:27.065433 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb16f268-1822-483a-bdca-87bafd95625e-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:47:27.065468 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:27.065464 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsmmz\" (UniqueName: \"kubernetes.io/projected/bb16f268-1822-483a-bdca-87bafd95625e-kube-api-access-bsmmz\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:47:27.065468 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:27.065474 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb16f268-1822-483a-bdca-87bafd95625e-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:47:27.065735 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:27.065483 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb16f268-1822-483a-bdca-87bafd95625e-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:47:27.172159 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:27.172121 4926 generic.go:356] "Generic (PLEG): container finished" podID="bb16f268-1822-483a-bdca-87bafd95625e" containerID="1ac366253e8b8b9f2a35e4ac19df45333893f878710b6218673569bfbec59de0" exitCode=0 Oct 30 21:47:27.172391 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:27.172173 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" Oct 30 21:47:27.172391 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:27.172206 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" event={"ID":"bb16f268-1822-483a-bdca-87bafd95625e","Type":"ContainerDied","Data":"1ac366253e8b8b9f2a35e4ac19df45333893f878710b6218673569bfbec59de0"} Oct 30 21:47:27.172391 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:27.172278 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5" event={"ID":"bb16f268-1822-483a-bdca-87bafd95625e","Type":"ContainerDied","Data":"e78a1b74667bb13a91d16771fc84ede3429bda5f146796cb2e795b532cf1875d"} Oct 30 21:47:27.172391 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:27.172308 4926 scope.go:122] "RemoveContainer" containerID="1ac366253e8b8b9f2a35e4ac19df45333893f878710b6218673569bfbec59de0" Oct 30 21:47:27.186857 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:27.186832 4926 scope.go:122] "RemoveContainer" containerID="1ac366253e8b8b9f2a35e4ac19df45333893f878710b6218673569bfbec59de0" Oct 30 21:47:27.187274 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:47:27.187247 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ac366253e8b8b9f2a35e4ac19df45333893f878710b6218673569bfbec59de0\": container with ID starting with 1ac366253e8b8b9f2a35e4ac19df45333893f878710b6218673569bfbec59de0 not found: ID does not exist" containerID="1ac366253e8b8b9f2a35e4ac19df45333893f878710b6218673569bfbec59de0" Oct 30 21:47:27.187402 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:27.187283 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac366253e8b8b9f2a35e4ac19df45333893f878710b6218673569bfbec59de0"} err="failed to get container status \"1ac366253e8b8b9f2a35e4ac19df45333893f878710b6218673569bfbec59de0\": rpc error: code = NotFound desc = could not find container \"1ac366253e8b8b9f2a35e4ac19df45333893f878710b6218673569bfbec59de0\": container with ID starting with 1ac366253e8b8b9f2a35e4ac19df45333893f878710b6218673569bfbec59de0 not found: ID does not exist" Oct 30 21:47:27.224869 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:27.224830 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5"] Oct 30 21:47:27.228631 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:27.228605 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-75c17d-openshift-default-7ff59874cf-wdsp5"] Oct 30 21:47:29.019693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:29.019654 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb16f268-1822-483a-bdca-87bafd95625e" path="/var/lib/kubelet/pods/bb16f268-1822-483a-bdca-87bafd95625e/volumes" Oct 30 21:47:37.486288 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:47:37.486244 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:47:40.955083 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:40.955022 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:47:46.431690 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:46.431622 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0de63b5c-8lrmf"] Oct 30 21:47:46.445126 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:46.445087 4926 cpu_manager.go:414] "containerMap: removing container" podUID="bb16f268-1822-483a-bdca-87bafd95625e" containerName="istio-proxy" Oct 30 21:47:46.445400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:46.445379 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb16f268-1822-483a-bdca-87bafd95625e" containerName="istio-proxy" Oct 30 21:47:46.445625 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:46.445610 4926 cpu_manager.go:414] "containerMap: removing container" podUID="acea6911-e45b-461a-b5b2-3ac1a5a637c7" containerName="revoke-keys" Oct 30 21:47:46.445767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:46.445741 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acea6911-e45b-461a-b5b2-3ac1a5a637c7" containerName="revoke-keys" Oct 30 21:47:46.446148 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:46.446126 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb16f268-1822-483a-bdca-87bafd95625e" containerName="istio-proxy" Oct 30 21:47:46.446285 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:46.446274 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="acea6911-e45b-461a-b5b2-3ac1a5a637c7" containerName="revoke-keys" Oct 30 21:47:46.468524 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:46.468491 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0de63b5c-8lrmf" Oct 30 21:47:46.475335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:46.475304 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0de63b5c-8lrmf"] Oct 30 21:47:46.485132 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:46.485099 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:47:46.495830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:46.495798 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:47:46.507142 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:46.507100 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/034be202-cb88-4f23-8433-bc1f7bea9573-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0de63b5c-8lrmf\" (UID: \"034be202-cb88-4f23-8433-bc1f7bea9573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0de63b5c-8lrmf" Oct 30 21:47:46.507310 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:46.507259 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bckdt\" (UniqueName: \"kubernetes.io/projected/034be202-cb88-4f23-8433-bc1f7bea9573-kube-api-access-bckdt\") pod \"maas-api-revoke-sub-0de63b5c-8lrmf\" (UID: \"034be202-cb88-4f23-8433-bc1f7bea9573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0de63b5c-8lrmf" Oct 30 21:47:46.507408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:46.507330 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/034be202-cb88-4f23-8433-bc1f7bea9573-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0de63b5c-8lrmf\" (UID: \"034be202-cb88-4f23-8433-bc1f7bea9573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0de63b5c-8lrmf" Oct 30 21:47:46.608618 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:46.608563 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bckdt\" (UniqueName: \"kubernetes.io/projected/034be202-cb88-4f23-8433-bc1f7bea9573-kube-api-access-bckdt\") pod \"maas-api-revoke-sub-0de63b5c-8lrmf\" (UID: \"034be202-cb88-4f23-8433-bc1f7bea9573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0de63b5c-8lrmf" Oct 30 21:47:46.608855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:46.608673 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/034be202-cb88-4f23-8433-bc1f7bea9573-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0de63b5c-8lrmf\" (UID: \"034be202-cb88-4f23-8433-bc1f7bea9573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0de63b5c-8lrmf" Oct 30 21:47:46.608932 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:46.608883 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/034be202-cb88-4f23-8433-bc1f7bea9573-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0de63b5c-8lrmf\" (UID: \"034be202-cb88-4f23-8433-bc1f7bea9573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0de63b5c-8lrmf" Oct 30 21:47:46.609770 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:46.609722 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/034be202-cb88-4f23-8433-bc1f7bea9573-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0de63b5c-8lrmf\" (UID: \"034be202-cb88-4f23-8433-bc1f7bea9573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0de63b5c-8lrmf" Oct 30 21:47:46.611813 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:46.611781 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/034be202-cb88-4f23-8433-bc1f7bea9573-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0de63b5c-8lrmf\" (UID: \"034be202-cb88-4f23-8433-bc1f7bea9573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0de63b5c-8lrmf" Oct 30 21:47:46.634881 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:46.634849 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bckdt\" (UniqueName: \"kubernetes.io/projected/034be202-cb88-4f23-8433-bc1f7bea9573-kube-api-access-bckdt\") pod \"maas-api-revoke-sub-0de63b5c-8lrmf\" (UID: \"034be202-cb88-4f23-8433-bc1f7bea9573\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0de63b5c-8lrmf" Oct 30 21:47:46.796815 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:46.796713 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0de63b5c-8lrmf" Oct 30 21:47:47.111135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:47.111101 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0de63b5c-8lrmf"] Oct 30 21:47:47.113083 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:47:47.113042 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod034be202_cb88_4f23_8433_bc1f7bea9573.slice/crio-812749fc0d4b02b819efd5ba15a50188ca269f207d8e5c1f0a42bdb144e269cd WatchSource:0}: Error finding container 812749fc0d4b02b819efd5ba15a50188ca269f207d8e5c1f0a42bdb144e269cd: Status 404 returned error can't find the container with id 812749fc0d4b02b819efd5ba15a50188ca269f207d8e5c1f0a42bdb144e269cd Oct 30 21:47:47.360502 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:47.360404 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0de63b5c-8lrmf" event={"ID":"034be202-cb88-4f23-8433-bc1f7bea9573","Type":"ContainerStarted","Data":"812749fc0d4b02b819efd5ba15a50188ca269f207d8e5c1f0a42bdb144e269cd"} Oct 30 21:47:48.371661 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:48.371553 4926 generic.go:356] "Generic (PLEG): container finished" podID="034be202-cb88-4f23-8433-bc1f7bea9573" containerID="2d032e663517763c305fddbb3a5ecdc7c0f1c09e844ff96a29be77de55fc4a1f" exitCode=0 Oct 30 21:47:48.372189 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:48.371659 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0de63b5c-8lrmf" event={"ID":"034be202-cb88-4f23-8433-bc1f7bea9573","Type":"ContainerDied","Data":"2d032e663517763c305fddbb3a5ecdc7c0f1c09e844ff96a29be77de55fc4a1f"} Oct 30 21:47:49.547972 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:49.547940 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0de63b5c-8lrmf" Oct 30 21:47:49.659106 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:49.658965 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/034be202-cb88-4f23-8433-bc1f7bea9573-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/034be202-cb88-4f23-8433-bc1f7bea9573-maas-api-openshift-ca\") pod \"034be202-cb88-4f23-8433-bc1f7bea9573\" (UID: \"034be202-cb88-4f23-8433-bc1f7bea9573\") " Oct 30 21:47:49.659310 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:49.659270 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/034be202-cb88-4f23-8433-bc1f7bea9573-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/034be202-cb88-4f23-8433-bc1f7bea9573-maas-api-xks-ca\") pod \"034be202-cb88-4f23-8433-bc1f7bea9573\" (UID: \"034be202-cb88-4f23-8433-bc1f7bea9573\") " Oct 30 21:47:49.659411 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:49.659394 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/034be202-cb88-4f23-8433-bc1f7bea9573-kube-api-access-bckdt\" (UniqueName: \"kubernetes.io/projected/034be202-cb88-4f23-8433-bc1f7bea9573-kube-api-access-bckdt\") pod \"034be202-cb88-4f23-8433-bc1f7bea9573\" (UID: \"034be202-cb88-4f23-8433-bc1f7bea9573\") " Oct 30 21:47:49.659478 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:49.659422 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/034be202-cb88-4f23-8433-bc1f7bea9573-maas-api-openshift-ca" pod "034be202-cb88-4f23-8433-bc1f7bea9573" (UID: "034be202-cb88-4f23-8433-bc1f7bea9573"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:47:49.659903 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:49.659881 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/034be202-cb88-4f23-8433-bc1f7bea9573-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:47:49.662125 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:49.662073 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/034be202-cb88-4f23-8433-bc1f7bea9573-maas-api-xks-ca" pod "034be202-cb88-4f23-8433-bc1f7bea9573" (UID: "034be202-cb88-4f23-8433-bc1f7bea9573"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:47:49.674190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:49.672454 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/034be202-cb88-4f23-8433-bc1f7bea9573-kube-api-access-bckdt" pod "034be202-cb88-4f23-8433-bc1f7bea9573" (UID: "034be202-cb88-4f23-8433-bc1f7bea9573"). InnerVolumeSpecName "kube-api-access-bckdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:47:49.760830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:49.760787 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/034be202-cb88-4f23-8433-bc1f7bea9573-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:47:49.760830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:49.760822 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bckdt\" (UniqueName: \"kubernetes.io/projected/034be202-cb88-4f23-8433-bc1f7bea9573-kube-api-access-bckdt\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:47:50.232326 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.232004 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56"] Oct 30 21:47:50.233354 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.232963 4926 cpu_manager.go:414] "containerMap: removing container" podUID="034be202-cb88-4f23-8433-bc1f7bea9573" containerName="revoke-keys" Oct 30 21:47:50.233354 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.232988 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="034be202-cb88-4f23-8433-bc1f7bea9573" containerName="revoke-keys" Oct 30 21:47:50.233354 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.233278 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="034be202-cb88-4f23-8433-bc1f7bea9573" containerName="revoke-keys" Oct 30 21:47:50.237668 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.237646 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.255093 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.255048 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-e60e16-openshift-default-dockercfg-q2jh2\"" Oct 30 21:47:50.269075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.268984 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-data\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.269264 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.269130 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-credential-socket\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.269264 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.269218 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-workload-socket\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.269264 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.269247 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-envoy\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.269455 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.269357 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-token\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.269455 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.269406 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfw4x\" (UniqueName: \"kubernetes.io/projected/9311b3e6-ceb2-49dd-ac53-dca966a540ce-kube-api-access-wfw4x\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.269455 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.269439 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-podinfo\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.269627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.269466 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istiod-ca-cert\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.269627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.269564 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-workload-certs\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.275326 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.275292 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56"] Oct 30 21:47:50.370522 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.370484 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfw4x\" (UniqueName: \"kubernetes.io/projected/9311b3e6-ceb2-49dd-ac53-dca966a540ce-kube-api-access-wfw4x\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.370746 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.370540 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-podinfo\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.370746 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.370572 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istiod-ca-cert\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.370746 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.370666 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-workload-certs\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.370746 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.370734 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-data\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.371006 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.370815 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-credential-socket\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.371006 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.370897 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-workload-socket\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.371006 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.370925 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-envoy\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.371205 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.371060 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-token\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.371356 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.371330 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-workload-certs\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.371690 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.371659 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istiod-ca-cert\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.371808 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.371745 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-data\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.371879 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.371825 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-workload-socket\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.372025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.371984 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-credential-socket\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.373725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.373664 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-podinfo\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.374574 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.374551 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-envoy\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.392325 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.392282 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0de63b5c-8lrmf" event={"ID":"034be202-cb88-4f23-8433-bc1f7bea9573","Type":"ContainerDied","Data":"812749fc0d4b02b819efd5ba15a50188ca269f207d8e5c1f0a42bdb144e269cd"} Oct 30 21:47:50.392506 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.392344 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="812749fc0d4b02b819efd5ba15a50188ca269f207d8e5c1f0a42bdb144e269cd" Oct 30 21:47:50.392506 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.392297 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0de63b5c-8lrmf" Oct 30 21:47:50.393006 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.392984 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfw4x\" (UniqueName: \"kubernetes.io/projected/9311b3e6-ceb2-49dd-ac53-dca966a540ce-kube-api-access-wfw4x\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.396703 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.396666 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-token\") pod \"e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.563111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.557108 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:50.936650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.936618 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56"] Oct 30 21:47:50.936936 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:47:50.936904 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9311b3e6_ceb2_49dd_ac53_dca966a540ce.slice/crio-f169d6b40b7a11131aced943c57cc06e5a02e472359954e96fb71c4710239570 WatchSource:0}: Error finding container f169d6b40b7a11131aced943c57cc06e5a02e472359954e96fb71c4710239570: Status 404 returned error can't find the container with id f169d6b40b7a11131aced943c57cc06e5a02e472359954e96fb71c4710239570 Oct 30 21:47:50.940368 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.940332 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:47:50.940482 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.940417 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:47:50.940482 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:50.940469 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:47:51.407568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:51.407441 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" event={"ID":"9311b3e6-ceb2-49dd-ac53-dca966a540ce","Type":"ContainerStarted","Data":"806e121a53bd76a9ee9392323b72eb5d0a3cf75d7ff232969ce4085224a4ba55"} Oct 30 21:47:51.407856 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:51.407821 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" event={"ID":"9311b3e6-ceb2-49dd-ac53-dca966a540ce","Type":"ContainerStarted","Data":"f169d6b40b7a11131aced943c57cc06e5a02e472359954e96fb71c4710239570"} Oct 30 21:47:51.455692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:51.454174 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" podStartSLOduration=1.454150318 podStartE2EDuration="1.454150318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:51.450719979 +0000 UTC m=+2858.571793724" watchObservedRunningTime="2026-10-30 21:47:51.454150318 +0000 UTC m=+2858.575224080" Oct 30 21:47:51.557499 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:51.557463 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:52.566621 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:52.566571 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:53.426063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:53.426013 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:53.427201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:53.427177 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:47:56.893774 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:56.893734 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv"] Oct 30 21:47:56.899545 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:56.899516 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv" Oct 30 21:47:56.927846 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:56.927802 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv"] Oct 30 21:47:56.931879 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:56.931067 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:47:56.931879 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:56.931172 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:47:56.985480 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:56.985428 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5882\" (UniqueName: \"kubernetes.io/projected/6efe2df4-eb09-4380-be6f-96538c9bea6a-kube-api-access-q5882\") pod \"maas-api-revoke-sub-2bcded31-fd2wv\" (UID: \"6efe2df4-eb09-4380-be6f-96538c9bea6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv" Oct 30 21:47:56.985683 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:56.985566 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6efe2df4-eb09-4380-be6f-96538c9bea6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bcded31-fd2wv\" (UID: \"6efe2df4-eb09-4380-be6f-96538c9bea6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv" Oct 30 21:47:56.985765 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:56.985749 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6efe2df4-eb09-4380-be6f-96538c9bea6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bcded31-fd2wv\" (UID: \"6efe2df4-eb09-4380-be6f-96538c9bea6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv" Oct 30 21:47:57.087643 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:57.087603 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6efe2df4-eb09-4380-be6f-96538c9bea6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bcded31-fd2wv\" (UID: \"6efe2df4-eb09-4380-be6f-96538c9bea6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv" Oct 30 21:47:57.087844 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:57.087778 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6efe2df4-eb09-4380-be6f-96538c9bea6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bcded31-fd2wv\" (UID: \"6efe2df4-eb09-4380-be6f-96538c9bea6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv" Oct 30 21:47:57.087915 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:57.087900 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5882\" (UniqueName: \"kubernetes.io/projected/6efe2df4-eb09-4380-be6f-96538c9bea6a-kube-api-access-q5882\") pod \"maas-api-revoke-sub-2bcded31-fd2wv\" (UID: \"6efe2df4-eb09-4380-be6f-96538c9bea6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv" Oct 30 21:47:57.088504 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:57.088474 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6efe2df4-eb09-4380-be6f-96538c9bea6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bcded31-fd2wv\" (UID: \"6efe2df4-eb09-4380-be6f-96538c9bea6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv" Oct 30 21:47:57.090339 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:57.090318 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6efe2df4-eb09-4380-be6f-96538c9bea6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bcded31-fd2wv\" (UID: \"6efe2df4-eb09-4380-be6f-96538c9bea6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv" Oct 30 21:47:57.107083 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:57.107051 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5882\" (UniqueName: \"kubernetes.io/projected/6efe2df4-eb09-4380-be6f-96538c9bea6a-kube-api-access-q5882\") pod \"maas-api-revoke-sub-2bcded31-fd2wv\" (UID: \"6efe2df4-eb09-4380-be6f-96538c9bea6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv" Oct 30 21:47:57.221023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:57.220978 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv" Oct 30 21:47:57.526757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:57.526723 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv"] Oct 30 21:47:57.528817 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:47:57.528784 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6efe2df4_eb09_4380_be6f_96538c9bea6a.slice/crio-3085ede0d6ad1bccd0a4441b9f92e6836b1989c2fe135829dfc5e15d78f34278 WatchSource:0}: Error finding container 3085ede0d6ad1bccd0a4441b9f92e6836b1989c2fe135829dfc5e15d78f34278: Status 404 returned error can't find the container with id 3085ede0d6ad1bccd0a4441b9f92e6836b1989c2fe135829dfc5e15d78f34278 Oct 30 21:47:58.555213 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:58.555075 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv" event={"ID":"6efe2df4-eb09-4380-be6f-96538c9bea6a","Type":"ContainerStarted","Data":"9349c3586c742fe2b4d934c14df798f951c97184cbd85124996bc58ba78dcf08"} Oct 30 21:47:58.555213 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:58.555148 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv" event={"ID":"6efe2df4-eb09-4380-be6f-96538c9bea6a","Type":"ContainerStarted","Data":"3085ede0d6ad1bccd0a4441b9f92e6836b1989c2fe135829dfc5e15d78f34278"} Oct 30 21:47:58.595309 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:58.595245 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv" podStartSLOduration=2.595228354 podStartE2EDuration="2.595228354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:58.59353776 +0000 UTC m=+2865.714611509" watchObservedRunningTime="2026-10-30 21:47:58.595228354 +0000 UTC m=+2865.716302113" Oct 30 21:47:58.623799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:58.623761 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9"] Oct 30 21:47:58.630253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:58.630218 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-cdcf9d-hl6t9"] Oct 30 21:47:59.019025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:59.018986 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acea6911-e45b-461a-b5b2-3ac1a5a637c7" path="/var/lib/kubelet/pods/acea6911-e45b-461a-b5b2-3ac1a5a637c7/volumes" Oct 30 21:47:59.566639 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:59.566606 4926 generic.go:356] "Generic (PLEG): container finished" podID="6efe2df4-eb09-4380-be6f-96538c9bea6a" containerID="9349c3586c742fe2b4d934c14df798f951c97184cbd85124996bc58ba78dcf08" exitCode=0 Oct 30 21:47:59.567126 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:47:59.566690 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv" event={"ID":"6efe2df4-eb09-4380-be6f-96538c9bea6a","Type":"ContainerDied","Data":"9349c3586c742fe2b4d934c14df798f951c97184cbd85124996bc58ba78dcf08"} Oct 30 21:48:00.752500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:00.752472 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv" Oct 30 21:48:00.810155 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:00.810121 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6efe2df4-eb09-4380-be6f-96538c9bea6a-kube-api-access-q5882\" (UniqueName: \"kubernetes.io/projected/6efe2df4-eb09-4380-be6f-96538c9bea6a-kube-api-access-q5882\") pod \"6efe2df4-eb09-4380-be6f-96538c9bea6a\" (UID: \"6efe2df4-eb09-4380-be6f-96538c9bea6a\") " Oct 30 21:48:00.810376 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:00.810229 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6efe2df4-eb09-4380-be6f-96538c9bea6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6efe2df4-eb09-4380-be6f-96538c9bea6a-maas-api-xks-ca\") pod \"6efe2df4-eb09-4380-be6f-96538c9bea6a\" (UID: \"6efe2df4-eb09-4380-be6f-96538c9bea6a\") " Oct 30 21:48:00.810376 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:00.810292 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6efe2df4-eb09-4380-be6f-96538c9bea6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6efe2df4-eb09-4380-be6f-96538c9bea6a-maas-api-openshift-ca\") pod \"6efe2df4-eb09-4380-be6f-96538c9bea6a\" (UID: \"6efe2df4-eb09-4380-be6f-96538c9bea6a\") " Oct 30 21:48:00.810960 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:00.810927 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6efe2df4-eb09-4380-be6f-96538c9bea6a-maas-api-openshift-ca" pod "6efe2df4-eb09-4380-be6f-96538c9bea6a" (UID: "6efe2df4-eb09-4380-be6f-96538c9bea6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:00.812976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:00.812942 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6efe2df4-eb09-4380-be6f-96538c9bea6a-maas-api-xks-ca" pod "6efe2df4-eb09-4380-be6f-96538c9bea6a" (UID: "6efe2df4-eb09-4380-be6f-96538c9bea6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:00.813382 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:00.813351 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6efe2df4-eb09-4380-be6f-96538c9bea6a-kube-api-access-q5882" pod "6efe2df4-eb09-4380-be6f-96538c9bea6a" (UID: "6efe2df4-eb09-4380-be6f-96538c9bea6a"). InnerVolumeSpecName "kube-api-access-q5882". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:00.912943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:00.912850 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6efe2df4-eb09-4380-be6f-96538c9bea6a-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:00.912943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:00.912889 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5882\" (UniqueName: \"kubernetes.io/projected/6efe2df4-eb09-4380-be6f-96538c9bea6a-kube-api-access-q5882\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:00.912943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:00.912904 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6efe2df4-eb09-4380-be6f-96538c9bea6a-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:01.588876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:01.588843 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv" Oct 30 21:48:01.588876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:01.588866 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv" event={"ID":"6efe2df4-eb09-4380-be6f-96538c9bea6a","Type":"ContainerDied","Data":"3085ede0d6ad1bccd0a4441b9f92e6836b1989c2fe135829dfc5e15d78f34278"} Oct 30 21:48:01.589212 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:01.588917 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3085ede0d6ad1bccd0a4441b9f92e6836b1989c2fe135829dfc5e15d78f34278" Oct 30 21:48:06.831367 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:06.831323 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn"] Oct 30 21:48:06.832328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:06.832299 4926 cpu_manager.go:414] "containerMap: removing container" podUID="6efe2df4-eb09-4380-be6f-96538c9bea6a" containerName="revoke-keys" Oct 30 21:48:06.832549 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:06.832525 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6efe2df4-eb09-4380-be6f-96538c9bea6a" containerName="revoke-keys" Oct 30 21:48:06.832808 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:06.832793 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="6efe2df4-eb09-4380-be6f-96538c9bea6a" containerName="revoke-keys" Oct 30 21:48:06.837441 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:06.837412 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn" Oct 30 21:48:06.853184 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:06.853154 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn"] Oct 30 21:48:06.887573 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:06.887508 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:48:06.887860 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:06.887834 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:48:07.012761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:07.012727 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea1d5120-11d7-4786-a2a4-3c551e7793ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn\" (UID: \"ea1d5120-11d7-4786-a2a4-3c551e7793ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn" Oct 30 21:48:07.012979 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:07.012940 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea1d5120-11d7-4786-a2a4-3c551e7793ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn\" (UID: \"ea1d5120-11d7-4786-a2a4-3c551e7793ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn" Oct 30 21:48:07.013067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:07.013017 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfjv9\" (UniqueName: \"kubernetes.io/projected/ea1d5120-11d7-4786-a2a4-3c551e7793ee-kube-api-access-wfjv9\") pod \"maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn\" (UID: \"ea1d5120-11d7-4786-a2a4-3c551e7793ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn" Oct 30 21:48:07.114279 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:07.114167 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea1d5120-11d7-4786-a2a4-3c551e7793ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn\" (UID: \"ea1d5120-11d7-4786-a2a4-3c551e7793ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn" Oct 30 21:48:07.114479 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:07.114351 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea1d5120-11d7-4786-a2a4-3c551e7793ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn\" (UID: \"ea1d5120-11d7-4786-a2a4-3c551e7793ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn" Oct 30 21:48:07.114479 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:07.114420 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfjv9\" (UniqueName: \"kubernetes.io/projected/ea1d5120-11d7-4786-a2a4-3c551e7793ee-kube-api-access-wfjv9\") pod \"maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn\" (UID: \"ea1d5120-11d7-4786-a2a4-3c551e7793ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn" Oct 30 21:48:07.115108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:07.115083 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea1d5120-11d7-4786-a2a4-3c551e7793ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn\" (UID: \"ea1d5120-11d7-4786-a2a4-3c551e7793ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn" Oct 30 21:48:07.116935 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:07.116910 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea1d5120-11d7-4786-a2a4-3c551e7793ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn\" (UID: \"ea1d5120-11d7-4786-a2a4-3c551e7793ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn" Oct 30 21:48:07.139597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:07.139559 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfjv9\" (UniqueName: \"kubernetes.io/projected/ea1d5120-11d7-4786-a2a4-3c551e7793ee-kube-api-access-wfjv9\") pod \"maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn\" (UID: \"ea1d5120-11d7-4786-a2a4-3c551e7793ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn" Oct 30 21:48:07.155743 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:07.155709 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn" Oct 30 21:48:07.480379 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:07.480339 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn"] Oct 30 21:48:07.486610 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:48:07.484825 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea1d5120_11d7_4786_a2a4_3c551e7793ee.slice/crio-d4af3f75b1edb3992a2869ad3e1d3cdfb185710d77436df164b19d59f1dda8a2 WatchSource:0}: Error finding container d4af3f75b1edb3992a2869ad3e1d3cdfb185710d77436df164b19d59f1dda8a2: Status 404 returned error can't find the container with id d4af3f75b1edb3992a2869ad3e1d3cdfb185710d77436df164b19d59f1dda8a2 Oct 30 21:48:07.497906 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:48:07.497872 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:48:07.678007 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:07.677971 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn" event={"ID":"ea1d5120-11d7-4786-a2a4-3c551e7793ee","Type":"ContainerStarted","Data":"d4af3f75b1edb3992a2869ad3e1d3cdfb185710d77436df164b19d59f1dda8a2"} Oct 30 21:48:08.690528 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:08.690488 4926 generic.go:356] "Generic (PLEG): container finished" podID="ea1d5120-11d7-4786-a2a4-3c551e7793ee" containerID="5bbc76c0fa41aeb4206546529ecec98a331fc77f348c8e7603b0dd590082f778" exitCode=0 Oct 30 21:48:08.691097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:08.690578 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn" event={"ID":"ea1d5120-11d7-4786-a2a4-3c551e7793ee","Type":"ContainerDied","Data":"5bbc76c0fa41aeb4206546529ecec98a331fc77f348c8e7603b0dd590082f778"} Oct 30 21:48:09.915235 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:09.915197 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn" Oct 30 21:48:10.058882 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:10.058775 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea1d5120-11d7-4786-a2a4-3c551e7793ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea1d5120-11d7-4786-a2a4-3c551e7793ee-maas-api-openshift-ca\") pod \"ea1d5120-11d7-4786-a2a4-3c551e7793ee\" (UID: \"ea1d5120-11d7-4786-a2a4-3c551e7793ee\") " Oct 30 21:48:10.059097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:10.058913 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea1d5120-11d7-4786-a2a4-3c551e7793ee-kube-api-access-wfjv9\" (UniqueName: \"kubernetes.io/projected/ea1d5120-11d7-4786-a2a4-3c551e7793ee-kube-api-access-wfjv9\") pod \"ea1d5120-11d7-4786-a2a4-3c551e7793ee\" (UID: \"ea1d5120-11d7-4786-a2a4-3c551e7793ee\") " Oct 30 21:48:10.059097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:10.058971 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea1d5120-11d7-4786-a2a4-3c551e7793ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea1d5120-11d7-4786-a2a4-3c551e7793ee-maas-api-xks-ca\") pod \"ea1d5120-11d7-4786-a2a4-3c551e7793ee\" (UID: \"ea1d5120-11d7-4786-a2a4-3c551e7793ee\") " Oct 30 21:48:10.059275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:10.059227 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea1d5120-11d7-4786-a2a4-3c551e7793ee-maas-api-openshift-ca" pod "ea1d5120-11d7-4786-a2a4-3c551e7793ee" (UID: "ea1d5120-11d7-4786-a2a4-3c551e7793ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:10.059673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:10.059643 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea1d5120-11d7-4786-a2a4-3c551e7793ee-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:10.061389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:10.061366 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea1d5120-11d7-4786-a2a4-3c551e7793ee-maas-api-xks-ca" pod "ea1d5120-11d7-4786-a2a4-3c551e7793ee" (UID: "ea1d5120-11d7-4786-a2a4-3c551e7793ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:10.061496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:10.061367 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea1d5120-11d7-4786-a2a4-3c551e7793ee-kube-api-access-wfjv9" pod "ea1d5120-11d7-4786-a2a4-3c551e7793ee" (UID: "ea1d5120-11d7-4786-a2a4-3c551e7793ee"). InnerVolumeSpecName "kube-api-access-wfjv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:10.160755 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:10.160716 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfjv9\" (UniqueName: \"kubernetes.io/projected/ea1d5120-11d7-4786-a2a4-3c551e7793ee-kube-api-access-wfjv9\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:10.160755 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:10.160752 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea1d5120-11d7-4786-a2a4-3c551e7793ee-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:10.718277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:10.718239 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn" Oct 30 21:48:10.718277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:10.718262 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn" event={"ID":"ea1d5120-11d7-4786-a2a4-3c551e7793ee","Type":"ContainerDied","Data":"d4af3f75b1edb3992a2869ad3e1d3cdfb185710d77436df164b19d59f1dda8a2"} Oct 30 21:48:10.718511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:10.718324 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4af3f75b1edb3992a2869ad3e1d3cdfb185710d77436df164b19d59f1dda8a2" Oct 30 21:48:12.689056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:12.680224 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl"] Oct 30 21:48:18.468505 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:18.468464 4926 scope.go:122] "RemoveContainer" containerID="9579c782b3ea91012b61c80778557e3e08148e50e5daac7bd78afbb498bc11e4" Oct 30 21:48:18.481045 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:18.481003 4926 scope.go:122] "RemoveContainer" containerID="912971045adfcaea085c5a1a8d98d3941f82ede5f65d19ed5e96128190af326d" Oct 30 21:48:20.306444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:20.306402 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj"] Oct 30 21:48:20.307124 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:20.306661 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" podUID="44921ce3-eab8-4250-84bb-3bf42ac953cb" containerName="maas-api" containerID="cri-o://35ebadad087eff4477feacb4db31ba2a32eac12417bdfb9299d8b25d556c6146" gracePeriod=30 Oct 30 21:48:20.733361 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:20.733325 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" Oct 30 21:48:20.856453 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:20.856361 4926 generic.go:356] "Generic (PLEG): container finished" podID="44921ce3-eab8-4250-84bb-3bf42ac953cb" containerID="35ebadad087eff4477feacb4db31ba2a32eac12417bdfb9299d8b25d556c6146" exitCode=0 Oct 30 21:48:20.856623 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:20.856463 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" Oct 30 21:48:20.856623 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:20.856485 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" event={"ID":"44921ce3-eab8-4250-84bb-3bf42ac953cb","Type":"ContainerDied","Data":"35ebadad087eff4477feacb4db31ba2a32eac12417bdfb9299d8b25d556c6146"} Oct 30 21:48:20.856623 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:20.856543 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj" event={"ID":"44921ce3-eab8-4250-84bb-3bf42ac953cb","Type":"ContainerDied","Data":"f33a376818e869c84bd1765ddeee551c7b46cd351550ba51574db3fec4f579bf"} Oct 30 21:48:20.856623 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:20.856569 4926 scope.go:122] "RemoveContainer" containerID="35ebadad087eff4477feacb4db31ba2a32eac12417bdfb9299d8b25d556c6146" Oct 30 21:48:20.872392 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:20.872279 4926 scope.go:122] "RemoveContainer" containerID="35ebadad087eff4477feacb4db31ba2a32eac12417bdfb9299d8b25d556c6146" Oct 30 21:48:20.872732 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:48:20.872704 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ebadad087eff4477feacb4db31ba2a32eac12417bdfb9299d8b25d556c6146\": container with ID starting with 35ebadad087eff4477feacb4db31ba2a32eac12417bdfb9299d8b25d556c6146 not found: ID does not exist" containerID="35ebadad087eff4477feacb4db31ba2a32eac12417bdfb9299d8b25d556c6146" Oct 30 21:48:20.872848 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:20.872743 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ebadad087eff4477feacb4db31ba2a32eac12417bdfb9299d8b25d556c6146"} err="failed to get container status \"35ebadad087eff4477feacb4db31ba2a32eac12417bdfb9299d8b25d556c6146\": rpc error: code = NotFound desc = could not find container \"35ebadad087eff4477feacb4db31ba2a32eac12417bdfb9299d8b25d556c6146\": container with ID starting with 35ebadad087eff4477feacb4db31ba2a32eac12417bdfb9299d8b25d556c6146 not found: ID does not exist" Oct 30 21:48:20.886943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:20.885654 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44921ce3-eab8-4250-84bb-3bf42ac953cb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/44921ce3-eab8-4250-84bb-3bf42ac953cb-metrics-certs\") pod \"44921ce3-eab8-4250-84bb-3bf42ac953cb\" (UID: \"44921ce3-eab8-4250-84bb-3bf42ac953cb\") " Oct 30 21:48:20.886943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:20.885784 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44921ce3-eab8-4250-84bb-3bf42ac953cb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44921ce3-eab8-4250-84bb-3bf42ac953cb-openshift-service-ca\") pod \"44921ce3-eab8-4250-84bb-3bf42ac953cb\" (UID: \"44921ce3-eab8-4250-84bb-3bf42ac953cb\") " Oct 30 21:48:20.886943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:20.885851 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44921ce3-eab8-4250-84bb-3bf42ac953cb-kube-api-access-58lrv\" (UniqueName: \"kubernetes.io/projected/44921ce3-eab8-4250-84bb-3bf42ac953cb-kube-api-access-58lrv\") pod \"44921ce3-eab8-4250-84bb-3bf42ac953cb\" (UID: \"44921ce3-eab8-4250-84bb-3bf42ac953cb\") " Oct 30 21:48:20.886943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:20.885993 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44921ce3-eab8-4250-84bb-3bf42ac953cb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44921ce3-eab8-4250-84bb-3bf42ac953cb-maas-api-tls\") pod \"44921ce3-eab8-4250-84bb-3bf42ac953cb\" (UID: \"44921ce3-eab8-4250-84bb-3bf42ac953cb\") " Oct 30 21:48:20.886943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:20.886362 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44921ce3-eab8-4250-84bb-3bf42ac953cb-openshift-service-ca" pod "44921ce3-eab8-4250-84bb-3bf42ac953cb" (UID: "44921ce3-eab8-4250-84bb-3bf42ac953cb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:20.886943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:20.886725 4926 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/44921ce3-eab8-4250-84bb-3bf42ac953cb-openshift-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:20.895193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:20.893804 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44921ce3-eab8-4250-84bb-3bf42ac953cb-metrics-certs" pod "44921ce3-eab8-4250-84bb-3bf42ac953cb" (UID: "44921ce3-eab8-4250-84bb-3bf42ac953cb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:20.895193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:20.895149 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44921ce3-eab8-4250-84bb-3bf42ac953cb-maas-api-tls" pod "44921ce3-eab8-4250-84bb-3bf42ac953cb" (UID: "44921ce3-eab8-4250-84bb-3bf42ac953cb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:20.902424 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:20.902394 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44921ce3-eab8-4250-84bb-3bf42ac953cb-kube-api-access-58lrv" pod "44921ce3-eab8-4250-84bb-3bf42ac953cb" (UID: "44921ce3-eab8-4250-84bb-3bf42ac953cb"). InnerVolumeSpecName "kube-api-access-58lrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:21.004059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:20.996321 4926 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/44921ce3-eab8-4250-84bb-3bf42ac953cb-metrics-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:21.004059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:20.996367 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58lrv\" (UniqueName: \"kubernetes.io/projected/44921ce3-eab8-4250-84bb-3bf42ac953cb-kube-api-access-58lrv\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:21.004059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:20.996383 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/44921ce3-eab8-4250-84bb-3bf42ac953cb-maas-api-tls\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:21.213692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:21.213646 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn"] Oct 30 21:48:21.221241 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:21.221202 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c3bbba-qsnfn"] Oct 30 21:48:21.262300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:21.262258 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj"] Oct 30 21:48:21.267563 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:21.267530 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c3bbba-75b6799d96-vlhrj"] Oct 30 21:48:23.018737 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:23.018692 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44921ce3-eab8-4250-84bb-3bf42ac953cb" path="/var/lib/kubelet/pods/44921ce3-eab8-4250-84bb-3bf42ac953cb/volumes" Oct 30 21:48:23.019506 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:23.019483 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea1d5120-11d7-4786-a2a4-3c551e7793ee" path="/var/lib/kubelet/pods/ea1d5120-11d7-4786-a2a4-3c551e7793ee/volumes" Oct 30 21:48:27.713553 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:27.713508 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" podUID="5f11288b-8b0d-4e07-8d5d-613780c8f390" containerName="main" containerID="cri-o://ba422981ad55bed82859bce250d699ab3b3be47a571f552515c271f8aaec20c1" gracePeriod=45 Oct 30 21:48:27.724762 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:27.724718 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" podUID="5f11288b-8b0d-4e07-8d5d-613780c8f390" containerName="main" probeResult="failure" output="Get \"https://10.132.0.218:8000/ready\": dial tcp 10.132.0.218:8000: connect: connection refused" Oct 30 21:48:27.943469 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:27.943425 4926 generic.go:356] "Generic (PLEG): container finished" podID="5f11288b-8b0d-4e07-8d5d-613780c8f390" containerID="ba422981ad55bed82859bce250d699ab3b3be47a571f552515c271f8aaec20c1" exitCode=0 Oct 30 21:48:27.943676 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:27.943487 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" event={"ID":"5f11288b-8b0d-4e07-8d5d-613780c8f390","Type":"ContainerDied","Data":"ba422981ad55bed82859bce250d699ab3b3be47a571f552515c271f8aaec20c1"} Oct 30 21:48:28.030108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:28.030075 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:48:28.065251 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:28.065216 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f11288b-8b0d-4e07-8d5d-613780c8f390-kube-api-access-vfbmn\" (UniqueName: \"kubernetes.io/projected/5f11288b-8b0d-4e07-8d5d-613780c8f390-kube-api-access-vfbmn\") pod \"5f11288b-8b0d-4e07-8d5d-613780c8f390\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " Oct 30 21:48:28.065449 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:28.065274 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-dshm\") pod \"5f11288b-8b0d-4e07-8d5d-613780c8f390\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " Oct 30 21:48:28.065449 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:28.065316 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f11288b-8b0d-4e07-8d5d-613780c8f390-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f11288b-8b0d-4e07-8d5d-613780c8f390-tls-certs\") pod \"5f11288b-8b0d-4e07-8d5d-613780c8f390\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " Oct 30 21:48:28.065449 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:28.065357 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-model-cache\") pod \"5f11288b-8b0d-4e07-8d5d-613780c8f390\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " Oct 30 21:48:28.065449 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:28.065426 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-home\" (UniqueName: \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-home\") pod \"5f11288b-8b0d-4e07-8d5d-613780c8f390\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " Oct 30 21:48:28.065657 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:28.065492 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-tmp-dir\") pod \"5f11288b-8b0d-4e07-8d5d-613780c8f390\" (UID: \"5f11288b-8b0d-4e07-8d5d-613780c8f390\") " Oct 30 21:48:28.065735 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:28.065690 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-model-cache" pod "5f11288b-8b0d-4e07-8d5d-613780c8f390" (UID: "5f11288b-8b0d-4e07-8d5d-613780c8f390"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:28.065819 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:28.065774 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-home" pod "5f11288b-8b0d-4e07-8d5d-613780c8f390" (UID: "5f11288b-8b0d-4e07-8d5d-613780c8f390"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:28.065885 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:28.065839 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-tmp-dir" pod "5f11288b-8b0d-4e07-8d5d-613780c8f390" (UID: "5f11288b-8b0d-4e07-8d5d-613780c8f390"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:28.066620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:28.066594 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:28.066620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:28.066622 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:28.066842 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:28.066635 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:28.067892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:28.067865 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-dshm" pod "5f11288b-8b0d-4e07-8d5d-613780c8f390" (UID: "5f11288b-8b0d-4e07-8d5d-613780c8f390"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:28.068022 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:28.067983 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f11288b-8b0d-4e07-8d5d-613780c8f390-kube-api-access-vfbmn" pod "5f11288b-8b0d-4e07-8d5d-613780c8f390" (UID: "5f11288b-8b0d-4e07-8d5d-613780c8f390"). InnerVolumeSpecName "kube-api-access-vfbmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:28.068022 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:28.067989 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f11288b-8b0d-4e07-8d5d-613780c8f390-tls-certs" pod "5f11288b-8b0d-4e07-8d5d-613780c8f390" (UID: "5f11288b-8b0d-4e07-8d5d-613780c8f390"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:28.168174 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:28.168132 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfbmn\" (UniqueName: \"kubernetes.io/projected/5f11288b-8b0d-4e07-8d5d-613780c8f390-kube-api-access-vfbmn\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:28.168174 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:28.168166 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f11288b-8b0d-4e07-8d5d-613780c8f390-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:28.168174 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:28.168176 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f11288b-8b0d-4e07-8d5d-613780c8f390-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:28.954285 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:28.954236 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" event={"ID":"5f11288b-8b0d-4e07-8d5d-613780c8f390","Type":"ContainerDied","Data":"8f542dd1cecef6ced857ae9c7eb3b1c729db6a6ef601c8cd52a1497841c1e55d"} Oct 30 21:48:28.954285 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:28.954282 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl" Oct 30 21:48:28.954852 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:28.954311 4926 scope.go:122] "RemoveContainer" containerID="ba422981ad55bed82859bce250d699ab3b3be47a571f552515c271f8aaec20c1" Oct 30 21:48:29.005115 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:29.005077 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl"] Oct 30 21:48:29.019022 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:29.018988 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-66cc54/auth-test-model-66cc54-kserve-6d8f968dfc-kdbxl"] Oct 30 21:48:30.249098 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:30.247765 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs"] Oct 30 21:48:30.249098 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:30.248049 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" podUID="c49c7689-1388-4442-8ea8-825da60fab98" containerName="istio-proxy" containerID="cri-o://ff397e6e0c9ebff2b2a24402f6d517d7744d7ab873981ac1c93dc040e6706fc2" gracePeriod=30 Oct 30 21:48:30.816616 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:30.816572 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz"] Oct 30 21:48:30.817456 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:30.817428 4926 cpu_manager.go:414] "containerMap: removing container" podUID="5f11288b-8b0d-4e07-8d5d-613780c8f390" containerName="main" Oct 30 21:48:30.817456 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:30.817452 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f11288b-8b0d-4e07-8d5d-613780c8f390" containerName="main" Oct 30 21:48:30.817456 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:30.817464 4926 cpu_manager.go:414] "containerMap: removing container" podUID="ea1d5120-11d7-4786-a2a4-3c551e7793ee" containerName="revoke-keys" Oct 30 21:48:30.817652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:30.817469 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea1d5120-11d7-4786-a2a4-3c551e7793ee" containerName="revoke-keys" Oct 30 21:48:30.817652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:30.817497 4926 cpu_manager.go:414] "containerMap: removing container" podUID="44921ce3-eab8-4250-84bb-3bf42ac953cb" containerName="maas-api" Oct 30 21:48:30.817652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:30.817505 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44921ce3-eab8-4250-84bb-3bf42ac953cb" containerName="maas-api" Oct 30 21:48:30.817652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:30.817604 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="44921ce3-eab8-4250-84bb-3bf42ac953cb" containerName="maas-api" Oct 30 21:48:30.817652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:30.817620 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f11288b-8b0d-4e07-8d5d-613780c8f390" containerName="main" Oct 30 21:48:30.817652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:30.817627 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea1d5120-11d7-4786-a2a4-3c551e7793ee" containerName="revoke-keys" Oct 30 21:48:30.822650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:30.822628 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz" Oct 30 21:48:30.840818 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:30.840787 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:48:30.840980 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:30.840951 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:48:30.841054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:30.840969 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz"] Oct 30 21:48:30.913437 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:30.913400 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz\" (UID: \"d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz" Oct 30 21:48:30.913612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:30.913534 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz\" (UID: \"d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz" Oct 30 21:48:30.913826 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:30.913805 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnhx7\" (UniqueName: \"kubernetes.io/projected/d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83-kube-api-access-jnhx7\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz\" (UID: \"d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz" Oct 30 21:48:31.014791 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:31.014621 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz\" (UID: \"d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz" Oct 30 21:48:31.014791 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:31.014683 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz\" (UID: \"d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz" Oct 30 21:48:31.015965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:31.014915 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnhx7\" (UniqueName: \"kubernetes.io/projected/d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83-kube-api-access-jnhx7\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz\" (UID: \"d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz" Oct 30 21:48:31.015965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:31.015856 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz\" (UID: \"d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz" Oct 30 21:48:31.017535 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:31.017508 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz\" (UID: \"d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz" Oct 30 21:48:31.021326 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:31.021299 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f11288b-8b0d-4e07-8d5d-613780c8f390" path="/var/lib/kubelet/pods/5f11288b-8b0d-4e07-8d5d-613780c8f390/volumes" Oct 30 21:48:31.036727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:31.036701 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnhx7\" (UniqueName: \"kubernetes.io/projected/d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83-kube-api-access-jnhx7\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz\" (UID: \"d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz" Oct 30 21:48:31.138002 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:31.137915 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz" Oct 30 21:48:31.459986 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:31.459946 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz"] Oct 30 21:48:31.460981 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:48:31.460952 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd68d9eb6_fa77_46e6_a2ee_13a66fbc3f83.slice/crio-4afb100a5a62cd605eeca2d8f6b47730157d5883642db426960284d93c6d0332 WatchSource:0}: Error finding container 4afb100a5a62cd605eeca2d8f6b47730157d5883642db426960284d93c6d0332: Status 404 returned error can't find the container with id 4afb100a5a62cd605eeca2d8f6b47730157d5883642db426960284d93c6d0332 Oct 30 21:48:31.994689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:31.994648 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz" event={"ID":"d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83","Type":"ContainerStarted","Data":"e5eef3cff581fa85d8fb26839d64aecf5c51ea538c0902532bc37c9078422299"} Oct 30 21:48:31.994894 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:31.994711 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz" event={"ID":"d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83","Type":"ContainerStarted","Data":"4afb100a5a62cd605eeca2d8f6b47730157d5883642db426960284d93c6d0332"} Oct 30 21:48:32.036939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:32.036884 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz" podStartSLOduration=2.036869355 podStartE2EDuration="2.036869355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:32.034373737 +0000 UTC m=+2899.155447484" watchObservedRunningTime="2026-10-30 21:48:32.036869355 +0000 UTC m=+2899.157943102" Oct 30 21:48:32.219120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:32.219082 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:48:32.219290 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:32.219141 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" podUID="c49c7689-1388-4442-8ea8-825da60fab98" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:48:33.004937 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:33.004895 4926 generic.go:356] "Generic (PLEG): container finished" podID="d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83" containerID="e5eef3cff581fa85d8fb26839d64aecf5c51ea538c0902532bc37c9078422299" exitCode=0 Oct 30 21:48:33.005388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:33.004962 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz" event={"ID":"d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83","Type":"ContainerDied","Data":"e5eef3cff581fa85d8fb26839d64aecf5c51ea538c0902532bc37c9078422299"} Oct 30 21:48:34.186228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:34.186197 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz" Oct 30 21:48:34.276184 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:34.276072 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83-maas-api-openshift-ca\") pod \"d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83\" (UID: \"d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83\") " Oct 30 21:48:34.276369 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:34.276191 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83-maas-api-xks-ca\") pod \"d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83\" (UID: \"d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83\") " Oct 30 21:48:34.276369 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:34.276351 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83-kube-api-access-jnhx7\" (UniqueName: \"kubernetes.io/projected/d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83-kube-api-access-jnhx7\") pod \"d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83\" (UID: \"d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83\") " Oct 30 21:48:34.276947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:34.276915 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83-maas-api-openshift-ca" pod "d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83" (UID: "d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:34.279909 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:34.279877 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83-maas-api-xks-ca" pod "d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83" (UID: "d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:34.311024 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:34.310941 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83-kube-api-access-jnhx7" pod "d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83" (UID: "d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83"). InnerVolumeSpecName "kube-api-access-jnhx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:34.379210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:34.379163 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:34.379210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:34.379207 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnhx7\" (UniqueName: \"kubernetes.io/projected/d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83-kube-api-access-jnhx7\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:34.379479 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:34.379223 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:35.037189 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.037136 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz" event={"ID":"d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83","Type":"ContainerDied","Data":"4afb100a5a62cd605eeca2d8f6b47730157d5883642db426960284d93c6d0332"} Oct 30 21:48:35.037445 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.037425 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4afb100a5a62cd605eeca2d8f6b47730157d5883642db426960284d93c6d0332" Oct 30 21:48:35.037549 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.037227 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz" Oct 30 21:48:35.539459 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.539430 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:48:35.597767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.597730 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c49c7689-1388-4442-8ea8-825da60fab98-istio-token\" (UniqueName: \"kubernetes.io/projected/c49c7689-1388-4442-8ea8-825da60fab98-istio-token\") pod \"c49c7689-1388-4442-8ea8-825da60fab98\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " Oct 30 21:48:35.597989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.597841 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-workload-certs\") pod \"c49c7689-1388-4442-8ea8-825da60fab98\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " Oct 30 21:48:35.597989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.597865 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-workload-socket\") pod \"c49c7689-1388-4442-8ea8-825da60fab98\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " Oct 30 21:48:35.597989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.597960 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-credential-socket\") pod \"c49c7689-1388-4442-8ea8-825da60fab98\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " Oct 30 21:48:35.598190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.598012 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c49c7689-1388-4442-8ea8-825da60fab98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49c7689-1388-4442-8ea8-825da60fab98-istio-podinfo\") pod \"c49c7689-1388-4442-8ea8-825da60fab98\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " Oct 30 21:48:35.598190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.598087 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c49c7689-1388-4442-8ea8-825da60fab98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49c7689-1388-4442-8ea8-825da60fab98-istiod-ca-cert\") pod \"c49c7689-1388-4442-8ea8-825da60fab98\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " Oct 30 21:48:35.598190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.598171 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c49c7689-1388-4442-8ea8-825da60fab98-kube-api-access-kktpw\" (UniqueName: \"kubernetes.io/projected/c49c7689-1388-4442-8ea8-825da60fab98-kube-api-access-kktpw\") pod \"c49c7689-1388-4442-8ea8-825da60fab98\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " Oct 30 21:48:35.598353 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.598217 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-istio-data\") pod \"c49c7689-1388-4442-8ea8-825da60fab98\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " Oct 30 21:48:35.598353 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.598273 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-workload-certs" pod "c49c7689-1388-4442-8ea8-825da60fab98" (UID: "c49c7689-1388-4442-8ea8-825da60fab98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:35.598353 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.598299 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-credential-socket" pod "c49c7689-1388-4442-8ea8-825da60fab98" (UID: "c49c7689-1388-4442-8ea8-825da60fab98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:35.598522 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.598385 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-istio-envoy\") pod \"c49c7689-1388-4442-8ea8-825da60fab98\" (UID: \"c49c7689-1388-4442-8ea8-825da60fab98\") " Oct 30 21:48:35.598586 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.598573 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c49c7689-1388-4442-8ea8-825da60fab98-istiod-ca-cert" pod "c49c7689-1388-4442-8ea8-825da60fab98" (UID: "c49c7689-1388-4442-8ea8-825da60fab98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:35.598927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.598837 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-workload-socket" pod "c49c7689-1388-4442-8ea8-825da60fab98" (UID: "c49c7689-1388-4442-8ea8-825da60fab98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:35.598927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.598883 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-istio-data" pod "c49c7689-1388-4442-8ea8-825da60fab98" (UID: "c49c7689-1388-4442-8ea8-825da60fab98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:35.599232 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.599206 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:35.599306 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.599234 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:35.599306 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.599248 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:35.599306 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.599262 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c49c7689-1388-4442-8ea8-825da60fab98-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:35.599306 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.599276 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:35.600712 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.600688 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-istio-envoy" pod "c49c7689-1388-4442-8ea8-825da60fab98" (UID: "c49c7689-1388-4442-8ea8-825da60fab98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:35.600908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.600883 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49c7689-1388-4442-8ea8-825da60fab98-istio-token" pod "c49c7689-1388-4442-8ea8-825da60fab98" (UID: "c49c7689-1388-4442-8ea8-825da60fab98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:35.601119 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.601088 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c49c7689-1388-4442-8ea8-825da60fab98-istio-podinfo" pod "c49c7689-1388-4442-8ea8-825da60fab98" (UID: "c49c7689-1388-4442-8ea8-825da60fab98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:48:35.601210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.601139 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c49c7689-1388-4442-8ea8-825da60fab98-kube-api-access-kktpw" pod "c49c7689-1388-4442-8ea8-825da60fab98" (UID: "c49c7689-1388-4442-8ea8-825da60fab98"). InnerVolumeSpecName "kube-api-access-kktpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:35.701207 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.701161 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c49c7689-1388-4442-8ea8-825da60fab98-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:35.701207 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.701206 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kktpw\" (UniqueName: \"kubernetes.io/projected/c49c7689-1388-4442-8ea8-825da60fab98-kube-api-access-kktpw\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:35.701482 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.701222 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c49c7689-1388-4442-8ea8-825da60fab98-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:35.701482 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:35.701236 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c49c7689-1388-4442-8ea8-825da60fab98-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:36.046839 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.046756 4926 generic.go:356] "Generic (PLEG): container finished" podID="c49c7689-1388-4442-8ea8-825da60fab98" containerID="ff397e6e0c9ebff2b2a24402f6d517d7744d7ab873981ac1c93dc040e6706fc2" exitCode=0 Oct 30 21:48:36.046839 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.046808 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" Oct 30 21:48:36.047157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.046836 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" event={"ID":"c49c7689-1388-4442-8ea8-825da60fab98","Type":"ContainerDied","Data":"ff397e6e0c9ebff2b2a24402f6d517d7744d7ab873981ac1c93dc040e6706fc2"} Oct 30 21:48:36.047157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.046892 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs" event={"ID":"c49c7689-1388-4442-8ea8-825da60fab98","Type":"ContainerDied","Data":"7f8407245e0cb68e7188389a1ce4aa3d6907a8a0aef61d3505761c3defee42dc"} Oct 30 21:48:36.047157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.046915 4926 scope.go:122] "RemoveContainer" containerID="ff397e6e0c9ebff2b2a24402f6d517d7744d7ab873981ac1c93dc040e6706fc2" Oct 30 21:48:36.059453 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.059425 4926 scope.go:122] "RemoveContainer" containerID="2b102897365236e348c1ba81bd40f11f89da75a791b377b6132bb10e0554394d" Oct 30 21:48:36.072503 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.072472 4926 scope.go:122] "RemoveContainer" containerID="ff397e6e0c9ebff2b2a24402f6d517d7744d7ab873981ac1c93dc040e6706fc2" Oct 30 21:48:36.072854 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:48:36.072832 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff397e6e0c9ebff2b2a24402f6d517d7744d7ab873981ac1c93dc040e6706fc2\": container with ID starting with ff397e6e0c9ebff2b2a24402f6d517d7744d7ab873981ac1c93dc040e6706fc2 not found: ID does not exist" containerID="ff397e6e0c9ebff2b2a24402f6d517d7744d7ab873981ac1c93dc040e6706fc2" Oct 30 21:48:36.072947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.072863 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff397e6e0c9ebff2b2a24402f6d517d7744d7ab873981ac1c93dc040e6706fc2"} err="failed to get container status \"ff397e6e0c9ebff2b2a24402f6d517d7744d7ab873981ac1c93dc040e6706fc2\": rpc error: code = NotFound desc = could not find container \"ff397e6e0c9ebff2b2a24402f6d517d7744d7ab873981ac1c93dc040e6706fc2\": container with ID starting with ff397e6e0c9ebff2b2a24402f6d517d7744d7ab873981ac1c93dc040e6706fc2 not found: ID does not exist" Oct 30 21:48:36.072947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.072885 4926 scope.go:122] "RemoveContainer" containerID="2b102897365236e348c1ba81bd40f11f89da75a791b377b6132bb10e0554394d" Oct 30 21:48:36.073179 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:48:36.073155 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b102897365236e348c1ba81bd40f11f89da75a791b377b6132bb10e0554394d\": container with ID starting with 2b102897365236e348c1ba81bd40f11f89da75a791b377b6132bb10e0554394d not found: ID does not exist" containerID="2b102897365236e348c1ba81bd40f11f89da75a791b377b6132bb10e0554394d" Oct 30 21:48:36.073243 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.073190 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b102897365236e348c1ba81bd40f11f89da75a791b377b6132bb10e0554394d"} err="failed to get container status \"2b102897365236e348c1ba81bd40f11f89da75a791b377b6132bb10e0554394d\": rpc error: code = NotFound desc = could not find container \"2b102897365236e348c1ba81bd40f11f89da75a791b377b6132bb10e0554394d\": container with ID starting with 2b102897365236e348c1ba81bd40f11f89da75a791b377b6132bb10e0554394d not found: ID does not exist" Oct 30 21:48:36.112358 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.112321 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs"] Oct 30 21:48:36.120894 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.120856 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-c3bbba-openshift-default-7d9c98b86c-lp5qs"] Oct 30 21:48:36.514798 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.514755 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t"] Oct 30 21:48:36.520081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.515670 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c49c7689-1388-4442-8ea8-825da60fab98" containerName="istio-proxy" Oct 30 21:48:36.520081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.515695 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c49c7689-1388-4442-8ea8-825da60fab98" containerName="istio-proxy" Oct 30 21:48:36.520081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.515707 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83" containerName="revoke-keys" Oct 30 21:48:36.520081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.515713 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83" containerName="revoke-keys" Oct 30 21:48:36.520081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.515739 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c49c7689-1388-4442-8ea8-825da60fab98" containerName="istio-proxy" Oct 30 21:48:36.520081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.515747 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c49c7689-1388-4442-8ea8-825da60fab98" containerName="istio-proxy" Oct 30 21:48:36.520081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.515909 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c49c7689-1388-4442-8ea8-825da60fab98" containerName="istio-proxy" Oct 30 21:48:36.520081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.515922 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83" containerName="revoke-keys" Oct 30 21:48:36.521626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.521603 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" Oct 30 21:48:36.538565 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.538535 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t"] Oct 30 21:48:36.539335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.539291 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-e60e16-serving-cert\"" Oct 30 21:48:36.613459 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.613411 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t\" (UID: \"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" Oct 30 21:48:36.614003 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.613494 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhbvp\" (UniqueName: \"kubernetes.io/projected/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-kube-api-access-qhbvp\") pod \"maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t\" (UID: \"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" Oct 30 21:48:36.614003 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.613616 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-metrics-certs\") pod \"maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t\" (UID: \"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" Oct 30 21:48:36.614003 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.613795 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t\" (UID: \"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" Oct 30 21:48:36.716302 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.716257 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhbvp\" (UniqueName: \"kubernetes.io/projected/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-kube-api-access-qhbvp\") pod \"maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t\" (UID: \"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" Oct 30 21:48:36.717063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.717014 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-metrics-certs\") pod \"maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t\" (UID: \"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" Oct 30 21:48:36.718026 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.718001 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t\" (UID: \"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" Oct 30 21:48:36.719061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.719022 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t\" (UID: \"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" Oct 30 21:48:36.719689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.718940 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t\" (UID: \"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" Oct 30 21:48:36.731651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.731617 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-metrics-certs\") pod \"maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t\" (UID: \"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" Oct 30 21:48:36.753768 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.753732 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t\" (UID: \"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" Oct 30 21:48:36.755535 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.755502 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhbvp\" (UniqueName: \"kubernetes.io/projected/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-kube-api-access-qhbvp\") pod \"maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t\" (UID: \"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" Oct 30 21:48:36.843321 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:36.843228 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" Oct 30 21:48:37.019656 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:37.019613 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c49c7689-1388-4442-8ea8-825da60fab98" path="/var/lib/kubelet/pods/c49c7689-1388-4442-8ea8-825da60fab98/volumes" Oct 30 21:48:37.169713 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:37.169680 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t"] Oct 30 21:48:37.171890 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:48:37.171849 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3c5ac1_e264_429e_8ebe_e51f6b5a0038.slice/crio-692a4db22d036c00466e9f05207dd1bc8febca5e49fdcbd6f623ed9f69e862be WatchSource:0}: Error finding container 692a4db22d036c00466e9f05207dd1bc8febca5e49fdcbd6f623ed9f69e862be: Status 404 returned error can't find the container with id 692a4db22d036c00466e9f05207dd1bc8febca5e49fdcbd6f623ed9f69e862be Oct 30 21:48:37.495583 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:48:37.495506 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:48:38.066555 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:38.066515 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" event={"ID":"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038","Type":"ContainerStarted","Data":"d3e730c2fab5b035d19fcbfec071bae76ada310c5fb082e7373398b3f8fe6644"} Oct 30 21:48:38.067013 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:38.066582 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" event={"ID":"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038","Type":"ContainerStarted","Data":"692a4db22d036c00466e9f05207dd1bc8febca5e49fdcbd6f623ed9f69e862be"} Oct 30 21:48:38.067013 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:38.066612 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" Oct 30 21:48:38.127647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:38.127590 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" podStartSLOduration=1.789912727 podStartE2EDuration="2.127572354s" totalImagesPullingTime="337.659627ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:38.126177792 +0000 UTC m=+2905.247251538" watchObservedRunningTime="2026-10-30 21:48:38.127572354 +0000 UTC m=+2905.248646097" Oct 30 21:48:44.082809 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:44.082774 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" Oct 30 21:48:45.240080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:45.240017 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz"] Oct 30 21:48:45.243999 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:45.243969 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-a4782a-b5fxz"] Oct 30 21:48:47.017601 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:47.017560 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83" path="/var/lib/kubelet/pods/d68d9eb6-fa77-46e6-a2ee-13a66fbc3f83/volumes" Oct 30 21:48:53.223457 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:53.223405 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4"] Oct 30 21:48:53.224111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:53.223700 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" podUID="278c1234-a1ec-4205-b4be-b0d567c6cd10" containerName="istio-proxy" containerID="cri-o://228edb2f405679d4d16d8959e2efaf079c924cefc538a937c49889272e3eb4fb" gracePeriod=30 Oct 30 21:48:53.859682 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:53.859644 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq"] Oct 30 21:48:53.860443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:53.860423 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c49c7689-1388-4442-8ea8-825da60fab98" containerName="istio-proxy" Oct 30 21:48:53.865412 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:53.865393 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq" Oct 30 21:48:53.881010 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:53.880981 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:48:53.881265 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:53.881150 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:48:53.885433 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:53.885396 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq"] Oct 30 21:48:54.026233 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:54.026185 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhwd9\" (UniqueName: \"kubernetes.io/projected/101555da-36c4-4f1c-9603-a7d76bf05c25-kube-api-access-vhwd9\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq\" (UID: \"101555da-36c4-4f1c-9603-a7d76bf05c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq" Oct 30 21:48:54.026416 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:54.026325 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101555da-36c4-4f1c-9603-a7d76bf05c25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq\" (UID: \"101555da-36c4-4f1c-9603-a7d76bf05c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq" Oct 30 21:48:54.026553 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:54.026537 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101555da-36c4-4f1c-9603-a7d76bf05c25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq\" (UID: \"101555da-36c4-4f1c-9603-a7d76bf05c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq" Oct 30 21:48:54.128175 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:54.128074 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101555da-36c4-4f1c-9603-a7d76bf05c25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq\" (UID: \"101555da-36c4-4f1c-9603-a7d76bf05c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq" Oct 30 21:48:54.128377 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:54.128198 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhwd9\" (UniqueName: \"kubernetes.io/projected/101555da-36c4-4f1c-9603-a7d76bf05c25-kube-api-access-vhwd9\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq\" (UID: \"101555da-36c4-4f1c-9603-a7d76bf05c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq" Oct 30 21:48:54.128377 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:54.128302 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101555da-36c4-4f1c-9603-a7d76bf05c25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq\" (UID: \"101555da-36c4-4f1c-9603-a7d76bf05c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq" Oct 30 21:48:54.128986 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:54.128959 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101555da-36c4-4f1c-9603-a7d76bf05c25-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq\" (UID: \"101555da-36c4-4f1c-9603-a7d76bf05c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq" Oct 30 21:48:54.131088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:54.131063 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101555da-36c4-4f1c-9603-a7d76bf05c25-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq\" (UID: \"101555da-36c4-4f1c-9603-a7d76bf05c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq" Oct 30 21:48:54.156350 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:54.156319 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhwd9\" (UniqueName: \"kubernetes.io/projected/101555da-36c4-4f1c-9603-a7d76bf05c25-kube-api-access-vhwd9\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq\" (UID: \"101555da-36c4-4f1c-9603-a7d76bf05c25\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq" Oct 30 21:48:54.183933 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:54.183884 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq" Oct 30 21:48:54.615844 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:54.615807 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq"] Oct 30 21:48:55.269275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:55.269230 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq" event={"ID":"101555da-36c4-4f1c-9603-a7d76bf05c25","Type":"ContainerStarted","Data":"8bd33a8449f1e531a73babe7741f087b78cf9c1ab39fdf4daf766c4c571c300f"} Oct 30 21:48:55.269507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:55.269307 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq" event={"ID":"101555da-36c4-4f1c-9603-a7d76bf05c25","Type":"ContainerStarted","Data":"28d9f68dfbeed11cf16420fda6f828efcbe4fe1af730d696dbefc46768750c3a"} Oct 30 21:48:55.307605 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:55.307532 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq" podStartSLOduration=2.307514388 podStartE2EDuration="2.307514388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:55.306096009 +0000 UTC m=+2922.427169756" watchObservedRunningTime="2026-10-30 21:48:55.307514388 +0000 UTC m=+2922.428588134" Oct 30 21:48:56.282970 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:56.282925 4926 generic.go:356] "Generic (PLEG): container finished" podID="101555da-36c4-4f1c-9603-a7d76bf05c25" containerID="8bd33a8449f1e531a73babe7741f087b78cf9c1ab39fdf4daf766c4c571c300f" exitCode=0 Oct 30 21:48:56.283520 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:56.282976 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq" event={"ID":"101555da-36c4-4f1c-9603-a7d76bf05c25","Type":"ContainerDied","Data":"8bd33a8449f1e531a73babe7741f087b78cf9c1ab39fdf4daf766c4c571c300f"} Oct 30 21:48:57.458889 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:57.458858 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq" Oct 30 21:48:57.585358 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:57.585252 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/101555da-36c4-4f1c-9603-a7d76bf05c25-kube-api-access-vhwd9\" (UniqueName: \"kubernetes.io/projected/101555da-36c4-4f1c-9603-a7d76bf05c25-kube-api-access-vhwd9\") pod \"101555da-36c4-4f1c-9603-a7d76bf05c25\" (UID: \"101555da-36c4-4f1c-9603-a7d76bf05c25\") " Oct 30 21:48:57.585557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:57.585447 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/101555da-36c4-4f1c-9603-a7d76bf05c25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101555da-36c4-4f1c-9603-a7d76bf05c25-maas-api-xks-ca\") pod \"101555da-36c4-4f1c-9603-a7d76bf05c25\" (UID: \"101555da-36c4-4f1c-9603-a7d76bf05c25\") " Oct 30 21:48:57.585557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:57.585490 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/101555da-36c4-4f1c-9603-a7d76bf05c25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101555da-36c4-4f1c-9603-a7d76bf05c25-maas-api-openshift-ca\") pod \"101555da-36c4-4f1c-9603-a7d76bf05c25\" (UID: \"101555da-36c4-4f1c-9603-a7d76bf05c25\") " Oct 30 21:48:57.585924 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:57.585896 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/101555da-36c4-4f1c-9603-a7d76bf05c25-maas-api-openshift-ca" pod "101555da-36c4-4f1c-9603-a7d76bf05c25" (UID: "101555da-36c4-4f1c-9603-a7d76bf05c25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:57.586121 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:57.586105 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/101555da-36c4-4f1c-9603-a7d76bf05c25-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:57.587511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:57.587486 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/101555da-36c4-4f1c-9603-a7d76bf05c25-kube-api-access-vhwd9" pod "101555da-36c4-4f1c-9603-a7d76bf05c25" (UID: "101555da-36c4-4f1c-9603-a7d76bf05c25"). InnerVolumeSpecName "kube-api-access-vhwd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:57.587997 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:57.587972 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/101555da-36c4-4f1c-9603-a7d76bf05c25-maas-api-xks-ca" pod "101555da-36c4-4f1c-9603-a7d76bf05c25" (UID: "101555da-36c4-4f1c-9603-a7d76bf05c25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:48:57.687143 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:57.687104 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/101555da-36c4-4f1c-9603-a7d76bf05c25-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:57.687143 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:57.687135 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhwd9\" (UniqueName: \"kubernetes.io/projected/101555da-36c4-4f1c-9603-a7d76bf05c25-kube-api-access-vhwd9\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:58.307390 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.307356 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq" Oct 30 21:48:58.307390 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.307383 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq" event={"ID":"101555da-36c4-4f1c-9603-a7d76bf05c25","Type":"ContainerDied","Data":"28d9f68dfbeed11cf16420fda6f828efcbe4fe1af730d696dbefc46768750c3a"} Oct 30 21:48:58.307664 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.307432 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28d9f68dfbeed11cf16420fda6f828efcbe4fe1af730d696dbefc46768750c3a" Oct 30 21:48:58.511956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.511884 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:48:58.602938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.601027 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-workload-socket\") pod \"278c1234-a1ec-4205-b4be-b0d567c6cd10\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " Oct 30 21:48:58.602938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.601115 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-workload-certs\") pod \"278c1234-a1ec-4205-b4be-b0d567c6cd10\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " Oct 30 21:48:58.602938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.601172 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-data\") pod \"278c1234-a1ec-4205-b4be-b0d567c6cd10\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " Oct 30 21:48:58.602938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.601211 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-credential-socket\") pod \"278c1234-a1ec-4205-b4be-b0d567c6cd10\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " Oct 30 21:48:58.602938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.601277 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/278c1234-a1ec-4205-b4be-b0d567c6cd10-kube-api-access-twpsd\" (UniqueName: \"kubernetes.io/projected/278c1234-a1ec-4205-b4be-b0d567c6cd10-kube-api-access-twpsd\") pod \"278c1234-a1ec-4205-b4be-b0d567c6cd10\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " Oct 30 21:48:58.602938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.601352 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-podinfo\") pod \"278c1234-a1ec-4205-b4be-b0d567c6cd10\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " Oct 30 21:48:58.602938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.601399 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-envoy\") pod \"278c1234-a1ec-4205-b4be-b0d567c6cd10\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " Oct 30 21:48:58.602938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.601436 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/278c1234-a1ec-4205-b4be-b0d567c6cd10-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278c1234-a1ec-4205-b4be-b0d567c6cd10-istiod-ca-cert\") pod \"278c1234-a1ec-4205-b4be-b0d567c6cd10\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " Oct 30 21:48:58.602938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.601547 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-token\" (UniqueName: \"kubernetes.io/projected/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-token\") pod \"278c1234-a1ec-4205-b4be-b0d567c6cd10\" (UID: \"278c1234-a1ec-4205-b4be-b0d567c6cd10\") " Oct 30 21:48:58.603654 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.603177 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-workload-certs" pod "278c1234-a1ec-4205-b4be-b0d567c6cd10" (UID: "278c1234-a1ec-4205-b4be-b0d567c6cd10"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:58.603654 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.603412 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-workload-socket" pod "278c1234-a1ec-4205-b4be-b0d567c6cd10" (UID: "278c1234-a1ec-4205-b4be-b0d567c6cd10"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:58.603654 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.603504 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-credential-socket" pod "278c1234-a1ec-4205-b4be-b0d567c6cd10" (UID: "278c1234-a1ec-4205-b4be-b0d567c6cd10"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:58.603833 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.603710 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-data" pod "278c1234-a1ec-4205-b4be-b0d567c6cd10" (UID: "278c1234-a1ec-4205-b4be-b0d567c6cd10"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:58.603833 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.603741 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/278c1234-a1ec-4205-b4be-b0d567c6cd10-istiod-ca-cert" pod "278c1234-a1ec-4205-b4be-b0d567c6cd10" (UID: "278c1234-a1ec-4205-b4be-b0d567c6cd10"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:48:58.624627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.624584 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-podinfo" pod "278c1234-a1ec-4205-b4be-b0d567c6cd10" (UID: "278c1234-a1ec-4205-b4be-b0d567c6cd10"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:48:58.624788 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.624602 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-envoy" pod "278c1234-a1ec-4205-b4be-b0d567c6cd10" (UID: "278c1234-a1ec-4205-b4be-b0d567c6cd10"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:48:58.624861 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.624833 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-token" pod "278c1234-a1ec-4205-b4be-b0d567c6cd10" (UID: "278c1234-a1ec-4205-b4be-b0d567c6cd10"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:58.626260 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.626225 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/278c1234-a1ec-4205-b4be-b0d567c6cd10-kube-api-access-twpsd" pod "278c1234-a1ec-4205-b4be-b0d567c6cd10" (UID: "278c1234-a1ec-4205-b4be-b0d567c6cd10"). InnerVolumeSpecName "kube-api-access-twpsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:48:58.703711 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.703669 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:58.703711 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.703710 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:58.703989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.703725 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:58.703989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.703738 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:58.703989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.703751 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:58.703989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.703765 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twpsd\" (UniqueName: \"kubernetes.io/projected/278c1234-a1ec-4205-b4be-b0d567c6cd10-kube-api-access-twpsd\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:58.703989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.703781 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:58.703989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.703794 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/278c1234-a1ec-4205-b4be-b0d567c6cd10-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:58.703989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:58.703805 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/278c1234-a1ec-4205-b4be-b0d567c6cd10-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:48:59.318842 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:59.318749 4926 generic.go:356] "Generic (PLEG): container finished" podID="278c1234-a1ec-4205-b4be-b0d567c6cd10" containerID="228edb2f405679d4d16d8959e2efaf079c924cefc538a937c49889272e3eb4fb" exitCode=0 Oct 30 21:48:59.318842 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:59.318812 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" Oct 30 21:48:59.318842 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:59.318819 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" event={"ID":"278c1234-a1ec-4205-b4be-b0d567c6cd10","Type":"ContainerDied","Data":"228edb2f405679d4d16d8959e2efaf079c924cefc538a937c49889272e3eb4fb"} Oct 30 21:48:59.319163 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:59.318877 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4" event={"ID":"278c1234-a1ec-4205-b4be-b0d567c6cd10","Type":"ContainerDied","Data":"6e9062339543ee54de343f2c0d232bae34464cd927922512ee2af84c302a4384"} Oct 30 21:48:59.319163 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:59.318899 4926 scope.go:122] "RemoveContainer" containerID="228edb2f405679d4d16d8959e2efaf079c924cefc538a937c49889272e3eb4fb" Oct 30 21:48:59.330498 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:59.330475 4926 scope.go:122] "RemoveContainer" containerID="7bd4150b5406693c32b80b057d6d0f3c529988d734aa4618dcbbcb5570d29dad" Oct 30 21:48:59.341849 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:59.341824 4926 scope.go:122] "RemoveContainer" containerID="228edb2f405679d4d16d8959e2efaf079c924cefc538a937c49889272e3eb4fb" Oct 30 21:48:59.342161 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:48:59.342135 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"228edb2f405679d4d16d8959e2efaf079c924cefc538a937c49889272e3eb4fb\": container with ID starting with 228edb2f405679d4d16d8959e2efaf079c924cefc538a937c49889272e3eb4fb not found: ID does not exist" containerID="228edb2f405679d4d16d8959e2efaf079c924cefc538a937c49889272e3eb4fb" Oct 30 21:48:59.342232 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:59.342170 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"228edb2f405679d4d16d8959e2efaf079c924cefc538a937c49889272e3eb4fb"} err="failed to get container status \"228edb2f405679d4d16d8959e2efaf079c924cefc538a937c49889272e3eb4fb\": rpc error: code = NotFound desc = could not find container \"228edb2f405679d4d16d8959e2efaf079c924cefc538a937c49889272e3eb4fb\": container with ID starting with 228edb2f405679d4d16d8959e2efaf079c924cefc538a937c49889272e3eb4fb not found: ID does not exist" Oct 30 21:48:59.342232 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:59.342188 4926 scope.go:122] "RemoveContainer" containerID="7bd4150b5406693c32b80b057d6d0f3c529988d734aa4618dcbbcb5570d29dad" Oct 30 21:48:59.342508 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:48:59.342483 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd4150b5406693c32b80b057d6d0f3c529988d734aa4618dcbbcb5570d29dad\": container with ID starting with 7bd4150b5406693c32b80b057d6d0f3c529988d734aa4618dcbbcb5570d29dad not found: ID does not exist" containerID="7bd4150b5406693c32b80b057d6d0f3c529988d734aa4618dcbbcb5570d29dad" Oct 30 21:48:59.342561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:59.342518 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd4150b5406693c32b80b057d6d0f3c529988d734aa4618dcbbcb5570d29dad"} err="failed to get container status \"7bd4150b5406693c32b80b057d6d0f3c529988d734aa4618dcbbcb5570d29dad\": rpc error: code = NotFound desc = could not find container \"7bd4150b5406693c32b80b057d6d0f3c529988d734aa4618dcbbcb5570d29dad\": container with ID starting with 7bd4150b5406693c32b80b057d6d0f3c529988d734aa4618dcbbcb5570d29dad not found: ID does not exist" Oct 30 21:48:59.387301 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:59.387258 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4"] Oct 30 21:48:59.398159 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:48:59.398123 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-a4782a-openshift-default-948cb9897-4z5w4"] Oct 30 21:49:01.018604 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:01.018567 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="278c1234-a1ec-4205-b4be-b0d567c6cd10" path="/var/lib/kubelet/pods/278c1234-a1ec-4205-b4be-b0d567c6cd10/volumes" Oct 30 21:49:06.406786 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.406738 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02d82ca1-kzlm7"] Oct 30 21:49:06.407746 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.407721 4926 cpu_manager.go:414] "containerMap: removing container" podUID="101555da-36c4-4f1c-9603-a7d76bf05c25" containerName="revoke-keys" Oct 30 21:49:06.407849 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.407748 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="101555da-36c4-4f1c-9603-a7d76bf05c25" containerName="revoke-keys" Oct 30 21:49:06.407849 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.407778 4926 cpu_manager.go:414] "containerMap: removing container" podUID="278c1234-a1ec-4205-b4be-b0d567c6cd10" containerName="istio-proxy" Oct 30 21:49:06.407849 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.407786 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="278c1234-a1ec-4205-b4be-b0d567c6cd10" containerName="istio-proxy" Oct 30 21:49:06.407849 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.407806 4926 cpu_manager.go:414] "containerMap: removing container" podUID="278c1234-a1ec-4205-b4be-b0d567c6cd10" containerName="istio-proxy" Oct 30 21:49:06.407849 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.407815 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="278c1234-a1ec-4205-b4be-b0d567c6cd10" containerName="istio-proxy" Oct 30 21:49:06.408185 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.407953 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="101555da-36c4-4f1c-9603-a7d76bf05c25" containerName="revoke-keys" Oct 30 21:49:06.408185 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.407967 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="278c1234-a1ec-4205-b4be-b0d567c6cd10" containerName="istio-proxy" Oct 30 21:49:06.408185 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.407978 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="278c1234-a1ec-4205-b4be-b0d567c6cd10" containerName="istio-proxy" Oct 30 21:49:06.413973 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.413946 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d82ca1-kzlm7" Oct 30 21:49:06.427100 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.427005 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02d82ca1-kzlm7"] Oct 30 21:49:06.432761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.432731 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:49:06.432761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.432746 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:49:06.523638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.523601 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70dc8202-7b65-4ef5-b096-b5c71680ea27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02d82ca1-kzlm7\" (UID: \"70dc8202-7b65-4ef5-b096-b5c71680ea27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d82ca1-kzlm7" Oct 30 21:49:06.523874 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.523792 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70dc8202-7b65-4ef5-b096-b5c71680ea27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02d82ca1-kzlm7\" (UID: \"70dc8202-7b65-4ef5-b096-b5c71680ea27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d82ca1-kzlm7" Oct 30 21:49:06.523941 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.523924 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx474\" (UniqueName: \"kubernetes.io/projected/70dc8202-7b65-4ef5-b096-b5c71680ea27-kube-api-access-dx474\") pod \"maas-api-revoke-sub-02d82ca1-kzlm7\" (UID: \"70dc8202-7b65-4ef5-b096-b5c71680ea27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d82ca1-kzlm7" Oct 30 21:49:06.625077 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.625017 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70dc8202-7b65-4ef5-b096-b5c71680ea27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02d82ca1-kzlm7\" (UID: \"70dc8202-7b65-4ef5-b096-b5c71680ea27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d82ca1-kzlm7" Oct 30 21:49:06.625280 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.625154 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70dc8202-7b65-4ef5-b096-b5c71680ea27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02d82ca1-kzlm7\" (UID: \"70dc8202-7b65-4ef5-b096-b5c71680ea27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d82ca1-kzlm7" Oct 30 21:49:06.625280 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.625257 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx474\" (UniqueName: \"kubernetes.io/projected/70dc8202-7b65-4ef5-b096-b5c71680ea27-kube-api-access-dx474\") pod \"maas-api-revoke-sub-02d82ca1-kzlm7\" (UID: \"70dc8202-7b65-4ef5-b096-b5c71680ea27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d82ca1-kzlm7" Oct 30 21:49:06.625914 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.625885 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70dc8202-7b65-4ef5-b096-b5c71680ea27-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02d82ca1-kzlm7\" (UID: \"70dc8202-7b65-4ef5-b096-b5c71680ea27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d82ca1-kzlm7" Oct 30 21:49:06.627618 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.627591 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70dc8202-7b65-4ef5-b096-b5c71680ea27-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02d82ca1-kzlm7\" (UID: \"70dc8202-7b65-4ef5-b096-b5c71680ea27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d82ca1-kzlm7" Oct 30 21:49:06.644555 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.644521 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx474\" (UniqueName: \"kubernetes.io/projected/70dc8202-7b65-4ef5-b096-b5c71680ea27-kube-api-access-dx474\") pod \"maas-api-revoke-sub-02d82ca1-kzlm7\" (UID: \"70dc8202-7b65-4ef5-b096-b5c71680ea27\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d82ca1-kzlm7" Oct 30 21:49:06.731022 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:06.730987 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d82ca1-kzlm7" Oct 30 21:49:07.092874 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:07.092804 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02d82ca1-kzlm7"] Oct 30 21:49:07.096597 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:49:07.096561 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70dc8202_7b65_4ef5_b096_b5c71680ea27.slice/crio-8d08a3af98a0f4b452d50e081433394b45e3fe159e740f19280dd1c539a2999f WatchSource:0}: Error finding container 8d08a3af98a0f4b452d50e081433394b45e3fe159e740f19280dd1c539a2999f: Status 404 returned error can't find the container with id 8d08a3af98a0f4b452d50e081433394b45e3fe159e740f19280dd1c539a2999f Oct 30 21:49:07.399624 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:07.399586 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d82ca1-kzlm7" event={"ID":"70dc8202-7b65-4ef5-b096-b5c71680ea27","Type":"ContainerStarted","Data":"8d08a3af98a0f4b452d50e081433394b45e3fe159e740f19280dd1c539a2999f"} Oct 30 21:49:07.489483 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:49:07.489442 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:49:07.980944 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:07.980903 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:49:08.410545 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:08.410452 4926 generic.go:356] "Generic (PLEG): container finished" podID="70dc8202-7b65-4ef5-b096-b5c71680ea27" containerID="8b7598fcd8b02d87bc0a34a9fc197846b5597528046fbc675e6df39774ba63cd" exitCode=0 Oct 30 21:49:08.410545 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:08.410501 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d82ca1-kzlm7" event={"ID":"70dc8202-7b65-4ef5-b096-b5c71680ea27","Type":"ContainerDied","Data":"8b7598fcd8b02d87bc0a34a9fc197846b5597528046fbc675e6df39774ba63cd"} Oct 30 21:49:09.649752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:09.649722 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d82ca1-kzlm7" Oct 30 21:49:09.842114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:09.837830 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70dc8202-7b65-4ef5-b096-b5c71680ea27-kube-api-access-dx474\" (UniqueName: \"kubernetes.io/projected/70dc8202-7b65-4ef5-b096-b5c71680ea27-kube-api-access-dx474\") pod \"70dc8202-7b65-4ef5-b096-b5c71680ea27\" (UID: \"70dc8202-7b65-4ef5-b096-b5c71680ea27\") " Oct 30 21:49:09.842114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:09.837889 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70dc8202-7b65-4ef5-b096-b5c71680ea27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70dc8202-7b65-4ef5-b096-b5c71680ea27-maas-api-xks-ca\") pod \"70dc8202-7b65-4ef5-b096-b5c71680ea27\" (UID: \"70dc8202-7b65-4ef5-b096-b5c71680ea27\") " Oct 30 21:49:09.842114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:09.837921 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70dc8202-7b65-4ef5-b096-b5c71680ea27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70dc8202-7b65-4ef5-b096-b5c71680ea27-maas-api-openshift-ca\") pod \"70dc8202-7b65-4ef5-b096-b5c71680ea27\" (UID: \"70dc8202-7b65-4ef5-b096-b5c71680ea27\") " Oct 30 21:49:09.858058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:09.856267 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70dc8202-7b65-4ef5-b096-b5c71680ea27-maas-api-openshift-ca" pod "70dc8202-7b65-4ef5-b096-b5c71680ea27" (UID: "70dc8202-7b65-4ef5-b096-b5c71680ea27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:09.958672 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:09.957948 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70dc8202-7b65-4ef5-b096-b5c71680ea27-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:49:09.962056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:09.959278 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70dc8202-7b65-4ef5-b096-b5c71680ea27-maas-api-xks-ca" pod "70dc8202-7b65-4ef5-b096-b5c71680ea27" (UID: "70dc8202-7b65-4ef5-b096-b5c71680ea27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:09.962056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:09.959606 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70dc8202-7b65-4ef5-b096-b5c71680ea27-kube-api-access-dx474" pod "70dc8202-7b65-4ef5-b096-b5c71680ea27" (UID: "70dc8202-7b65-4ef5-b096-b5c71680ea27"). InnerVolumeSpecName "kube-api-access-dx474". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:10.061082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:10.059472 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx474\" (UniqueName: \"kubernetes.io/projected/70dc8202-7b65-4ef5-b096-b5c71680ea27-kube-api-access-dx474\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:49:10.061082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:10.059518 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70dc8202-7b65-4ef5-b096-b5c71680ea27-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:49:10.428486 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:10.428445 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d82ca1-kzlm7" event={"ID":"70dc8202-7b65-4ef5-b096-b5c71680ea27","Type":"ContainerDied","Data":"8d08a3af98a0f4b452d50e081433394b45e3fe159e740f19280dd1c539a2999f"} Oct 30 21:49:10.428725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:10.428510 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d08a3af98a0f4b452d50e081433394b45e3fe159e740f19280dd1c539a2999f" Oct 30 21:49:10.428725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:10.428467 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02d82ca1-kzlm7" Oct 30 21:49:16.727984 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:16.727933 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04a24cc5-qc9x7"] Oct 30 21:49:16.728794 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:16.728767 4926 cpu_manager.go:414] "containerMap: removing container" podUID="70dc8202-7b65-4ef5-b096-b5c71680ea27" containerName="revoke-keys" Oct 30 21:49:16.728794 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:16.728793 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70dc8202-7b65-4ef5-b096-b5c71680ea27" containerName="revoke-keys" Oct 30 21:49:16.729028 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:16.729013 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="70dc8202-7b65-4ef5-b096-b5c71680ea27" containerName="revoke-keys" Oct 30 21:49:16.732576 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:16.732553 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a24cc5-qc9x7" Oct 30 21:49:16.751916 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:16.751881 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04a24cc5-qc9x7"] Oct 30 21:49:16.757715 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:16.757677 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:49:16.757901 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:16.757745 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:49:16.793097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:16.792919 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09de0d7b-9001-49f4-bd49-7901f130b546-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04a24cc5-qc9x7\" (UID: \"09de0d7b-9001-49f4-bd49-7901f130b546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a24cc5-qc9x7" Oct 30 21:49:16.793731 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:16.793410 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09de0d7b-9001-49f4-bd49-7901f130b546-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04a24cc5-qc9x7\" (UID: \"09de0d7b-9001-49f4-bd49-7901f130b546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a24cc5-qc9x7" Oct 30 21:49:16.793731 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:16.793469 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngdfj\" (UniqueName: \"kubernetes.io/projected/09de0d7b-9001-49f4-bd49-7901f130b546-kube-api-access-ngdfj\") pod \"maas-api-revoke-sub-04a24cc5-qc9x7\" (UID: \"09de0d7b-9001-49f4-bd49-7901f130b546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a24cc5-qc9x7" Oct 30 21:49:16.895760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:16.894854 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09de0d7b-9001-49f4-bd49-7901f130b546-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04a24cc5-qc9x7\" (UID: \"09de0d7b-9001-49f4-bd49-7901f130b546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a24cc5-qc9x7" Oct 30 21:49:16.895760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:16.894927 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngdfj\" (UniqueName: \"kubernetes.io/projected/09de0d7b-9001-49f4-bd49-7901f130b546-kube-api-access-ngdfj\") pod \"maas-api-revoke-sub-04a24cc5-qc9x7\" (UID: \"09de0d7b-9001-49f4-bd49-7901f130b546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a24cc5-qc9x7" Oct 30 21:49:16.895760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:16.895133 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09de0d7b-9001-49f4-bd49-7901f130b546-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04a24cc5-qc9x7\" (UID: \"09de0d7b-9001-49f4-bd49-7901f130b546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a24cc5-qc9x7" Oct 30 21:49:16.896997 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:16.896943 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09de0d7b-9001-49f4-bd49-7901f130b546-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-04a24cc5-qc9x7\" (UID: \"09de0d7b-9001-49f4-bd49-7901f130b546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a24cc5-qc9x7" Oct 30 21:49:16.899161 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:16.899113 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09de0d7b-9001-49f4-bd49-7901f130b546-maas-api-xks-ca\") pod \"maas-api-revoke-sub-04a24cc5-qc9x7\" (UID: \"09de0d7b-9001-49f4-bd49-7901f130b546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a24cc5-qc9x7" Oct 30 21:49:16.915721 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:16.915685 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngdfj\" (UniqueName: \"kubernetes.io/projected/09de0d7b-9001-49f4-bd49-7901f130b546-kube-api-access-ngdfj\") pod \"maas-api-revoke-sub-04a24cc5-qc9x7\" (UID: \"09de0d7b-9001-49f4-bd49-7901f130b546\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a24cc5-qc9x7" Oct 30 21:49:17.051818 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:17.051725 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a24cc5-qc9x7" Oct 30 21:49:17.384858 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:49:17.384794 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09de0d7b_9001_49f4_bd49_7901f130b546.slice/crio-17045bff781ad276d001f19faec6ce9ef829374d4c57852a45cbc7507126f7e0 WatchSource:0}: Error finding container 17045bff781ad276d001f19faec6ce9ef829374d4c57852a45cbc7507126f7e0: Status 404 returned error can't find the container with id 17045bff781ad276d001f19faec6ce9ef829374d4c57852a45cbc7507126f7e0 Oct 30 21:49:17.390991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:17.390945 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04a24cc5-qc9x7"] Oct 30 21:49:17.506802 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:17.506765 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a24cc5-qc9x7" event={"ID":"09de0d7b-9001-49f4-bd49-7901f130b546","Type":"ContainerStarted","Data":"17045bff781ad276d001f19faec6ce9ef829374d4c57852a45cbc7507126f7e0"} Oct 30 21:49:18.528820 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:18.528735 4926 generic.go:356] "Generic (PLEG): container finished" podID="09de0d7b-9001-49f4-bd49-7901f130b546" containerID="5764c27f407dd176a4e72bde98b490d4ac40e9de9b060703e81a9706bc8d5698" exitCode=0 Oct 30 21:49:18.529395 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:18.528832 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a24cc5-qc9x7" event={"ID":"09de0d7b-9001-49f4-bd49-7901f130b546","Type":"ContainerDied","Data":"5764c27f407dd176a4e72bde98b490d4ac40e9de9b060703e81a9706bc8d5698"} Oct 30 21:49:19.538109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:19.538074 4926 generic.go:356] "Generic (PLEG): container finished" podID="9311b3e6-ceb2-49dd-ac53-dca966a540ce" containerID="806e121a53bd76a9ee9392323b72eb5d0a3cf75d7ff232969ce4085224a4ba55" exitCode=0 Oct 30 21:49:19.538708 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:19.538152 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" event={"ID":"9311b3e6-ceb2-49dd-ac53-dca966a540ce","Type":"ContainerDied","Data":"806e121a53bd76a9ee9392323b72eb5d0a3cf75d7ff232969ce4085224a4ba55"} Oct 30 21:49:19.538708 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:19.538607 4926 scope.go:122] "RemoveContainer" containerID="806e121a53bd76a9ee9392323b72eb5d0a3cf75d7ff232969ce4085224a4ba55" Oct 30 21:49:19.540782 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:19.540745 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:49:19.540898 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:19.540841 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:49:19.540898 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:19.540889 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:49:19.765289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:19.765102 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a24cc5-qc9x7" Oct 30 21:49:19.839716 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:19.839613 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09de0d7b-9001-49f4-bd49-7901f130b546-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09de0d7b-9001-49f4-bd49-7901f130b546-maas-api-openshift-ca\") pod \"09de0d7b-9001-49f4-bd49-7901f130b546\" (UID: \"09de0d7b-9001-49f4-bd49-7901f130b546\") " Oct 30 21:49:19.839716 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:19.839669 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09de0d7b-9001-49f4-bd49-7901f130b546-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09de0d7b-9001-49f4-bd49-7901f130b546-maas-api-xks-ca\") pod \"09de0d7b-9001-49f4-bd49-7901f130b546\" (UID: \"09de0d7b-9001-49f4-bd49-7901f130b546\") " Oct 30 21:49:19.840010 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:19.839974 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09de0d7b-9001-49f4-bd49-7901f130b546-kube-api-access-ngdfj\" (UniqueName: \"kubernetes.io/projected/09de0d7b-9001-49f4-bd49-7901f130b546-kube-api-access-ngdfj\") pod \"09de0d7b-9001-49f4-bd49-7901f130b546\" (UID: \"09de0d7b-9001-49f4-bd49-7901f130b546\") " Oct 30 21:49:19.840130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:19.840104 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09de0d7b-9001-49f4-bd49-7901f130b546-maas-api-openshift-ca" pod "09de0d7b-9001-49f4-bd49-7901f130b546" (UID: "09de0d7b-9001-49f4-bd49-7901f130b546"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:19.840575 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:19.840546 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09de0d7b-9001-49f4-bd49-7901f130b546-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:49:19.842952 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:19.842907 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09de0d7b-9001-49f4-bd49-7901f130b546-kube-api-access-ngdfj" pod "09de0d7b-9001-49f4-bd49-7901f130b546" (UID: "09de0d7b-9001-49f4-bd49-7901f130b546"). InnerVolumeSpecName "kube-api-access-ngdfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:19.842952 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:19.842928 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09de0d7b-9001-49f4-bd49-7901f130b546-maas-api-xks-ca" pod "09de0d7b-9001-49f4-bd49-7901f130b546" (UID: "09de0d7b-9001-49f4-bd49-7901f130b546"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:19.941811 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:19.941776 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngdfj\" (UniqueName: \"kubernetes.io/projected/09de0d7b-9001-49f4-bd49-7901f130b546-kube-api-access-ngdfj\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:49:19.941811 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:19.941809 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09de0d7b-9001-49f4-bd49-7901f130b546-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:49:20.033830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:20.033791 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9"] Oct 30 21:49:20.038230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:20.038198 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-86dd7365-krmb9"] Oct 30 21:49:20.546873 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:20.546842 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a24cc5-qc9x7" Oct 30 21:49:20.547417 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:20.546882 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-04a24cc5-qc9x7" event={"ID":"09de0d7b-9001-49f4-bd49-7901f130b546","Type":"ContainerDied","Data":"17045bff781ad276d001f19faec6ce9ef829374d4c57852a45cbc7507126f7e0"} Oct 30 21:49:20.547417 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:20.546931 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17045bff781ad276d001f19faec6ce9ef829374d4c57852a45cbc7507126f7e0" Oct 30 21:49:20.549195 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:20.549172 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" event={"ID":"9311b3e6-ceb2-49dd-ac53-dca966a540ce","Type":"ContainerStarted","Data":"d050528c0af19898255956875d89a95870917075d4b657bffa3f09c6479a0c17"} Oct 30 21:49:20.557601 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:20.557580 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:49:20.559138 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:20.559115 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 30 21:49:20.559240 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:20.559171 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" podUID="9311b3e6-ceb2-49dd-ac53-dca966a540ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 30 21:49:21.019550 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:21.019504 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cfebe73-9386-47d7-9d93-46e0f38b5b71" path="/var/lib/kubelet/pods/9cfebe73-9386-47d7-9d93-46e0f38b5b71/volumes" Oct 30 21:49:21.558345 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:21.558304 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 30 21:49:21.558862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:21.558357 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" podUID="9311b3e6-ceb2-49dd-ac53-dca966a540ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 30 21:49:22.557748 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:22.557713 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 30 21:49:22.557962 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:22.557789 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" podUID="9311b3e6-ceb2-49dd-ac53-dca966a540ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 30 21:49:23.426079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:23.426041 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:49:23.558594 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:23.558557 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 30 21:49:23.558785 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:23.558616 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" podUID="9311b3e6-ceb2-49dd-ac53-dca966a540ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 30 21:49:24.557734 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:24.557688 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 30 21:49:24.558336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:24.557755 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" podUID="9311b3e6-ceb2-49dd-ac53-dca966a540ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 30 21:49:25.558991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:25.558954 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body= Oct 30 21:49:25.559438 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:25.559027 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" podUID="9311b3e6-ceb2-49dd-ac53-dca966a540ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" Oct 30 21:49:26.600673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:26.600630 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:49:26.602821 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:26.602785 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:49:30.055792 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:30.055757 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23a1a48b-222bl"] Oct 30 21:49:30.080895 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:30.080855 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23a1a48b-222bl"] Oct 30 21:49:31.018409 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:31.018376 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54d68ed2-dfbd-4aea-98ec-d182e78c2267" path="/var/lib/kubelet/pods/54d68ed2-dfbd-4aea-98ec-d182e78c2267/volumes" Oct 30 21:49:37.488171 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:49:37.488131 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:49:38.550070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:38.550010 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz"] Oct 30 21:49:38.550654 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:38.550636 4926 cpu_manager.go:414] "containerMap: removing container" podUID="09de0d7b-9001-49f4-bd49-7901f130b546" containerName="revoke-keys" Oct 30 21:49:38.550706 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:38.550655 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09de0d7b-9001-49f4-bd49-7901f130b546" containerName="revoke-keys" Oct 30 21:49:38.550885 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:38.550862 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="09de0d7b-9001-49f4-bd49-7901f130b546" containerName="revoke-keys" Oct 30 21:49:38.556538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:38.556516 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz" Oct 30 21:49:38.578753 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:38.578718 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz"] Oct 30 21:49:38.582982 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:38.582953 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f9a8ed\"" Oct 30 21:49:38.583196 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:38.583159 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f9a8ed-dockercfg-5wvsd\"" Oct 30 21:49:38.668356 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:38.668315 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/856a4b63-41f8-42de-ad43-2a2ffe41dfc4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz\" (UID: \"856a4b63-41f8-42de-ad43-2a2ffe41dfc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz" Oct 30 21:49:38.668824 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:38.668791 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j5s2\" (UniqueName: \"kubernetes.io/projected/856a4b63-41f8-42de-ad43-2a2ffe41dfc4-kube-api-access-9j5s2\") pod \"payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz\" (UID: \"856a4b63-41f8-42de-ad43-2a2ffe41dfc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz" Oct 30 21:49:38.822173 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:38.822079 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j5s2\" (UniqueName: \"kubernetes.io/projected/856a4b63-41f8-42de-ad43-2a2ffe41dfc4-kube-api-access-9j5s2\") pod \"payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz\" (UID: \"856a4b63-41f8-42de-ad43-2a2ffe41dfc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz" Oct 30 21:49:38.824063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:38.824020 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/856a4b63-41f8-42de-ad43-2a2ffe41dfc4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz\" (UID: \"856a4b63-41f8-42de-ad43-2a2ffe41dfc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz" Oct 30 21:49:38.824283 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:38.824227 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/856a4b63-41f8-42de-ad43-2a2ffe41dfc4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz\" (UID: \"856a4b63-41f8-42de-ad43-2a2ffe41dfc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz" Oct 30 21:49:38.846140 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:38.843821 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j5s2\" (UniqueName: \"kubernetes.io/projected/856a4b63-41f8-42de-ad43-2a2ffe41dfc4-kube-api-access-9j5s2\") pod \"payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz\" (UID: \"856a4b63-41f8-42de-ad43-2a2ffe41dfc4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz" Oct 30 21:49:38.872356 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:38.872310 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz" Oct 30 21:49:39.196821 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:39.196782 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz"] Oct 30 21:49:39.197643 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:49:39.197611 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856a4b63_41f8_42de_ad43_2a2ffe41dfc4.slice/crio-01a1f2d4835dd3bef24efc0e0193d4b30f326753a432bfce2d56dfaf3bbde721 WatchSource:0}: Error finding container 01a1f2d4835dd3bef24efc0e0193d4b30f326753a432bfce2d56dfaf3bbde721: Status 404 returned error can't find the container with id 01a1f2d4835dd3bef24efc0e0193d4b30f326753a432bfce2d56dfaf3bbde721 Oct 30 21:49:39.836562 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:39.836520 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz" event={"ID":"856a4b63-41f8-42de-ad43-2a2ffe41dfc4","Type":"ContainerStarted","Data":"535e6631f775189893492b3c472afd72dcdbef93bcf07ea9fb51120ca14a3621"} Oct 30 21:49:39.837067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:39.836584 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz" event={"ID":"856a4b63-41f8-42de-ad43-2a2ffe41dfc4","Type":"ContainerStarted","Data":"01a1f2d4835dd3bef24efc0e0193d4b30f326753a432bfce2d56dfaf3bbde721"} Oct 30 21:49:39.837067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:39.836607 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz" Oct 30 21:49:39.891761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:39.882585 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz" podStartSLOduration=1.882562369 podStartE2EDuration="1.882562369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:39.87297575 +0000 UTC m=+2966.994049497" watchObservedRunningTime="2026-10-30 21:49:39.882562369 +0000 UTC m=+2967.003636123" Oct 30 21:49:43.451899 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:43.451855 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq"] Oct 30 21:49:43.463214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:43.463181 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-75c17d-c5qzq"] Oct 30 21:49:45.018769 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:45.018719 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="101555da-36c4-4f1c-9603-a7d76bf05c25" path="/var/lib/kubelet/pods/101555da-36c4-4f1c-9603-a7d76bf05c25/volumes" Oct 30 21:49:50.590424 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:50.590378 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz"] Oct 30 21:49:50.849882 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:50.849797 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz" Oct 30 21:49:52.776688 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:52.776639 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx"] Oct 30 21:49:52.782215 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:52.782186 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx" Oct 30 21:49:52.799404 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:52.799370 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx"] Oct 30 21:49:52.805334 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:52.805302 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:49:52.805554 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:52.805402 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:49:52.918079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:52.918022 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v69w7\" (UniqueName: \"kubernetes.io/projected/19fcacf5-242f-4417-9092-45bd2fd7d781-kube-api-access-v69w7\") pod \"maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx\" (UID: \"19fcacf5-242f-4417-9092-45bd2fd7d781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx" Oct 30 21:49:52.918287 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:52.918242 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19fcacf5-242f-4417-9092-45bd2fd7d781-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx\" (UID: \"19fcacf5-242f-4417-9092-45bd2fd7d781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx" Oct 30 21:49:52.918448 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:52.918423 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19fcacf5-242f-4417-9092-45bd2fd7d781-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx\" (UID: \"19fcacf5-242f-4417-9092-45bd2fd7d781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx" Oct 30 21:49:53.020262 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:53.020218 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v69w7\" (UniqueName: \"kubernetes.io/projected/19fcacf5-242f-4417-9092-45bd2fd7d781-kube-api-access-v69w7\") pod \"maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx\" (UID: \"19fcacf5-242f-4417-9092-45bd2fd7d781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx" Oct 30 21:49:53.020467 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:53.020314 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19fcacf5-242f-4417-9092-45bd2fd7d781-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx\" (UID: \"19fcacf5-242f-4417-9092-45bd2fd7d781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx" Oct 30 21:49:53.020467 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:53.020383 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19fcacf5-242f-4417-9092-45bd2fd7d781-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx\" (UID: \"19fcacf5-242f-4417-9092-45bd2fd7d781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx" Oct 30 21:49:53.021080 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:53.021049 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19fcacf5-242f-4417-9092-45bd2fd7d781-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx\" (UID: \"19fcacf5-242f-4417-9092-45bd2fd7d781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx" Oct 30 21:49:53.022804 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:53.022780 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19fcacf5-242f-4417-9092-45bd2fd7d781-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx\" (UID: \"19fcacf5-242f-4417-9092-45bd2fd7d781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx" Oct 30 21:49:53.039620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:53.039542 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v69w7\" (UniqueName: \"kubernetes.io/projected/19fcacf5-242f-4417-9092-45bd2fd7d781-kube-api-access-v69w7\") pod \"maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx\" (UID: \"19fcacf5-242f-4417-9092-45bd2fd7d781\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx" Oct 30 21:49:53.102365 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:53.102327 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx" Oct 30 21:49:53.410518 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:49:53.410481 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19fcacf5_242f_4417_9092_45bd2fd7d781.slice/crio-b72b1be793adce07c55b2026e26d2e8137503ba164f6e9da926c8abb42ac1bc9 WatchSource:0}: Error finding container b72b1be793adce07c55b2026e26d2e8137503ba164f6e9da926c8abb42ac1bc9: Status 404 returned error can't find the container with id b72b1be793adce07c55b2026e26d2e8137503ba164f6e9da926c8abb42ac1bc9 Oct 30 21:49:53.410702 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:53.410641 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx"] Oct 30 21:49:54.005866 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:54.005822 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx" event={"ID":"19fcacf5-242f-4417-9092-45bd2fd7d781","Type":"ContainerStarted","Data":"1e38251da8d3bf3179c15870c641c964f0f5ffb5751501718f8d9e25cd939d2d"} Oct 30 21:49:54.006429 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:54.005891 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx" event={"ID":"19fcacf5-242f-4417-9092-45bd2fd7d781","Type":"ContainerStarted","Data":"b72b1be793adce07c55b2026e26d2e8137503ba164f6e9da926c8abb42ac1bc9"} Oct 30 21:49:54.046340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:54.046242 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx" podStartSLOduration=2.046226967 podStartE2EDuration="2.046226967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:54.045255717 +0000 UTC m=+2981.166329462" watchObservedRunningTime="2026-10-30 21:49:54.046226967 +0000 UTC m=+2981.167300712" Oct 30 21:49:55.016786 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:55.016748 4926 generic.go:356] "Generic (PLEG): container finished" podID="19fcacf5-242f-4417-9092-45bd2fd7d781" containerID="1e38251da8d3bf3179c15870c641c964f0f5ffb5751501718f8d9e25cd939d2d" exitCode=0 Oct 30 21:49:55.018836 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:55.018807 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx" event={"ID":"19fcacf5-242f-4417-9092-45bd2fd7d781","Type":"ContainerDied","Data":"1e38251da8d3bf3179c15870c641c964f0f5ffb5751501718f8d9e25cd939d2d"} Oct 30 21:49:56.208553 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:56.208522 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx" Oct 30 21:49:56.368454 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:56.368354 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19fcacf5-242f-4417-9092-45bd2fd7d781-kube-api-access-v69w7\" (UniqueName: \"kubernetes.io/projected/19fcacf5-242f-4417-9092-45bd2fd7d781-kube-api-access-v69w7\") pod \"19fcacf5-242f-4417-9092-45bd2fd7d781\" (UID: \"19fcacf5-242f-4417-9092-45bd2fd7d781\") " Oct 30 21:49:56.368645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:56.368521 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19fcacf5-242f-4417-9092-45bd2fd7d781-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19fcacf5-242f-4417-9092-45bd2fd7d781-maas-api-openshift-ca\") pod \"19fcacf5-242f-4417-9092-45bd2fd7d781\" (UID: \"19fcacf5-242f-4417-9092-45bd2fd7d781\") " Oct 30 21:49:56.368738 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:56.368670 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19fcacf5-242f-4417-9092-45bd2fd7d781-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19fcacf5-242f-4417-9092-45bd2fd7d781-maas-api-xks-ca\") pod \"19fcacf5-242f-4417-9092-45bd2fd7d781\" (UID: \"19fcacf5-242f-4417-9092-45bd2fd7d781\") " Oct 30 21:49:56.369120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:56.369089 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19fcacf5-242f-4417-9092-45bd2fd7d781-maas-api-openshift-ca" pod "19fcacf5-242f-4417-9092-45bd2fd7d781" (UID: "19fcacf5-242f-4417-9092-45bd2fd7d781"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:49:56.369610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:56.369584 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19fcacf5-242f-4417-9092-45bd2fd7d781-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:49:56.371430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:56.371369 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19fcacf5-242f-4417-9092-45bd2fd7d781-kube-api-access-v69w7" pod "19fcacf5-242f-4417-9092-45bd2fd7d781" (UID: "19fcacf5-242f-4417-9092-45bd2fd7d781"). InnerVolumeSpecName "kube-api-access-v69w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:49:56.384787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:56.384744 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19fcacf5-242f-4417-9092-45bd2fd7d781-maas-api-xks-ca" pod "19fcacf5-242f-4417-9092-45bd2fd7d781" (UID: "19fcacf5-242f-4417-9092-45bd2fd7d781"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:49:56.471869 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:56.471828 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v69w7\" (UniqueName: \"kubernetes.io/projected/19fcacf5-242f-4417-9092-45bd2fd7d781-kube-api-access-v69w7\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:49:56.472150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:56.472129 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19fcacf5-242f-4417-9092-45bd2fd7d781-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:49:57.034369 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:57.034335 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx" Oct 30 21:49:57.034544 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:57.034367 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx" event={"ID":"19fcacf5-242f-4417-9092-45bd2fd7d781","Type":"ContainerDied","Data":"b72b1be793adce07c55b2026e26d2e8137503ba164f6e9da926c8abb42ac1bc9"} Oct 30 21:49:57.034544 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:49:57.034422 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b72b1be793adce07c55b2026e26d2e8137503ba164f6e9da926c8abb42ac1bc9" Oct 30 21:50:01.034306 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:01.034267 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-641b550a-rlr56"] Oct 30 21:50:01.045404 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:01.045372 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-641b550a-rlr56"] Oct 30 21:50:03.018705 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:03.018671 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f" path="/var/lib/kubelet/pods/66a5f0b5-aff3-43d6-b25a-acbcc3ab1f0f/volumes" Oct 30 21:50:03.176489 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:03.176451 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm"] Oct 30 21:50:04.970703 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:04.970641 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:50:05.612893 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:05.612848 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" podUID="8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d" containerName="main" containerID="cri-o://089abf3af1fe3d1a8b01b1363f1819c97b4d38efc8a414fa6d48e75b683b139d" gracePeriod=45 Oct 30 21:50:05.964284 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:05.964250 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:50:06.046111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.045740 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-kube-api-access-bkcmm\" (UniqueName: \"kubernetes.io/projected/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-kube-api-access-bkcmm\") pod \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " Oct 30 21:50:06.046111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.045824 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-tmp-dir\") pod \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " Oct 30 21:50:06.046111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.045885 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-tls-certs\" (UniqueName: \"kubernetes.io/secret/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-tls-certs\") pod \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " Oct 30 21:50:06.046733 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.046134 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-model-cache\") pod \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " Oct 30 21:50:06.046733 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.046282 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-home\" (UniqueName: \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-home\") pod \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " Oct 30 21:50:06.046733 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.046327 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-dshm\") pod \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\" (UID: \"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d\") " Oct 30 21:50:06.046733 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.046715 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-tmp-dir" pod "8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d" (UID: "8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:06.049806 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.047067 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-home" pod "8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d" (UID: "8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:06.049806 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.048719 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:50:06.049806 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.048746 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:50:06.050237 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.047337 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-model-cache" pod "8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d" (UID: "8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:06.052607 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.052575 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-dshm" pod "8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d" (UID: "8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:06.052887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.052855 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-kube-api-access-bkcmm" pod "8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d" (UID: "8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d"). InnerVolumeSpecName "kube-api-access-bkcmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:06.053019 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.052995 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-tls-certs" pod "8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d" (UID: "8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:06.117512 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.117473 4926 generic.go:356] "Generic (PLEG): container finished" podID="8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d" containerID="089abf3af1fe3d1a8b01b1363f1819c97b4d38efc8a414fa6d48e75b683b139d" exitCode=0 Oct 30 21:50:06.117726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.117599 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" event={"ID":"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d","Type":"ContainerDied","Data":"089abf3af1fe3d1a8b01b1363f1819c97b4d38efc8a414fa6d48e75b683b139d"} Oct 30 21:50:06.117726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.117627 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" event={"ID":"8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d","Type":"ContainerDied","Data":"4f8b82e6f3501d1b0c14ea4514e9c19ce537934da3e11e105b089a72aa2c367f"} Oct 30 21:50:06.117726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.117645 4926 scope.go:122] "RemoveContainer" containerID="089abf3af1fe3d1a8b01b1363f1819c97b4d38efc8a414fa6d48e75b683b139d" Oct 30 21:50:06.117908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.117770 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" Oct 30 21:50:06.139864 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.139834 4926 scope.go:122] "RemoveContainer" containerID="089abf3af1fe3d1a8b01b1363f1819c97b4d38efc8a414fa6d48e75b683b139d" Oct 30 21:50:06.140282 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:50:06.140257 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"089abf3af1fe3d1a8b01b1363f1819c97b4d38efc8a414fa6d48e75b683b139d\": container with ID starting with 089abf3af1fe3d1a8b01b1363f1819c97b4d38efc8a414fa6d48e75b683b139d not found: ID does not exist" containerID="089abf3af1fe3d1a8b01b1363f1819c97b4d38efc8a414fa6d48e75b683b139d" Oct 30 21:50:06.140366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.140293 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"089abf3af1fe3d1a8b01b1363f1819c97b4d38efc8a414fa6d48e75b683b139d"} err="failed to get container status \"089abf3af1fe3d1a8b01b1363f1819c97b4d38efc8a414fa6d48e75b683b139d\": rpc error: code = NotFound desc = could not find container \"089abf3af1fe3d1a8b01b1363f1819c97b4d38efc8a414fa6d48e75b683b139d\": container with ID starting with 089abf3af1fe3d1a8b01b1363f1819c97b4d38efc8a414fa6d48e75b683b139d not found: ID does not exist" Oct 30 21:50:06.149832 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.149801 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkcmm\" (UniqueName: \"kubernetes.io/projected/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-kube-api-access-bkcmm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:50:06.149832 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.149830 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:50:06.149973 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.149842 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:50:06.149973 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.149851 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:50:06.179659 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.179619 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz"] Oct 30 21:50:06.198761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.198724 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz"] Oct 30 21:50:06.946528 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:06.946477 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-da52fd/rate-test-model-da52fd-kserve-76fff44757-qwchz" podUID="8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.222:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:50:07.020121 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:07.020080 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d" path="/var/lib/kubelet/pods/8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d/volumes" Oct 30 21:50:07.487907 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:50:07.487869 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:50:08.177117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:08.177042 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm" podUID="73ef630c-e7d0-4b76-b194-792276f55a67" containerName="payload-processing" containerID="cri-o://571d580e9ca5ccb32eecb779c674cf7269dbd93e9151569fd5643f054e703a02" gracePeriod=25 Oct 30 21:50:12.283178 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:12.283136 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm" podUID="73ef630c-e7d0-4b76-b194-792276f55a67" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:50:13.762282 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:13.762247 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:50:13.762767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:13.762311 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:50:15.906637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:15.906599 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v"] Oct 30 21:50:15.907238 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:15.907220 4926 cpu_manager.go:414] "containerMap: removing container" podUID="19fcacf5-242f-4417-9092-45bd2fd7d781" containerName="revoke-keys" Oct 30 21:50:15.907300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:15.907239 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19fcacf5-242f-4417-9092-45bd2fd7d781" containerName="revoke-keys" Oct 30 21:50:15.907300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:15.907266 4926 cpu_manager.go:414] "containerMap: removing container" podUID="8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d" containerName="main" Oct 30 21:50:15.907300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:15.907273 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d" containerName="main" Oct 30 21:50:15.907400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:15.907374 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="8baf34ec-5ec8-4bb1-b17a-8b6a3f39b80d" containerName="main" Oct 30 21:50:15.907400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:15.907391 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="19fcacf5-242f-4417-9092-45bd2fd7d781" containerName="revoke-keys" Oct 30 21:50:15.910909 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:15.910885 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v" Oct 30 21:50:15.929653 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:15.929615 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7"] Oct 30 21:50:15.934286 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:15.934246 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v"] Oct 30 21:50:15.934452 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:15.934360 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7" Oct 30 21:50:15.936047 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:15.936015 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:50:15.936214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:15.936196 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:50:15.954921 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:15.954887 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7"] Oct 30 21:50:15.996821 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:15.996756 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3920bbc-2a92-438d-ab8a-d4e076d3602d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-161e2a60-78b7v\" (UID: \"b3920bbc-2a92-438d-ab8a-d4e076d3602d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v" Oct 30 21:50:15.997084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:15.996948 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qnzh\" (UniqueName: \"kubernetes.io/projected/b3920bbc-2a92-438d-ab8a-d4e076d3602d-kube-api-access-5qnzh\") pod \"maas-api-revoke-sub-161e2a60-78b7v\" (UID: \"b3920bbc-2a92-438d-ab8a-d4e076d3602d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v" Oct 30 21:50:15.997188 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:15.997113 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3920bbc-2a92-438d-ab8a-d4e076d3602d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-161e2a60-78b7v\" (UID: \"b3920bbc-2a92-438d-ab8a-d4e076d3602d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v" Oct 30 21:50:16.099113 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:16.099027 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d83901a8-dbdf-4385-8e0f-af7d2fa13ccf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ccec6d1-gfql7\" (UID: \"d83901a8-dbdf-4385-8e0f-af7d2fa13ccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7" Oct 30 21:50:16.099462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:16.099431 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3920bbc-2a92-438d-ab8a-d4e076d3602d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-161e2a60-78b7v\" (UID: \"b3920bbc-2a92-438d-ab8a-d4e076d3602d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v" Oct 30 21:50:16.099619 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:16.099568 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3920bbc-2a92-438d-ab8a-d4e076d3602d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-161e2a60-78b7v\" (UID: \"b3920bbc-2a92-438d-ab8a-d4e076d3602d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v" Oct 30 21:50:16.099619 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:16.099605 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d83901a8-dbdf-4385-8e0f-af7d2fa13ccf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ccec6d1-gfql7\" (UID: \"d83901a8-dbdf-4385-8e0f-af7d2fa13ccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7" Oct 30 21:50:16.100004 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:16.099975 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6l9n\" (UniqueName: \"kubernetes.io/projected/d83901a8-dbdf-4385-8e0f-af7d2fa13ccf-kube-api-access-z6l9n\") pod \"maas-api-revoke-sub-9ccec6d1-gfql7\" (UID: \"d83901a8-dbdf-4385-8e0f-af7d2fa13ccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7" Oct 30 21:50:16.100355 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:16.100331 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qnzh\" (UniqueName: \"kubernetes.io/projected/b3920bbc-2a92-438d-ab8a-d4e076d3602d-kube-api-access-5qnzh\") pod \"maas-api-revoke-sub-161e2a60-78b7v\" (UID: \"b3920bbc-2a92-438d-ab8a-d4e076d3602d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v" Oct 30 21:50:16.100553 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:16.100529 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3920bbc-2a92-438d-ab8a-d4e076d3602d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-161e2a60-78b7v\" (UID: \"b3920bbc-2a92-438d-ab8a-d4e076d3602d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v" Oct 30 21:50:16.102640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:16.102618 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3920bbc-2a92-438d-ab8a-d4e076d3602d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-161e2a60-78b7v\" (UID: \"b3920bbc-2a92-438d-ab8a-d4e076d3602d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v" Oct 30 21:50:16.118794 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:16.118765 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qnzh\" (UniqueName: \"kubernetes.io/projected/b3920bbc-2a92-438d-ab8a-d4e076d3602d-kube-api-access-5qnzh\") pod \"maas-api-revoke-sub-161e2a60-78b7v\" (UID: \"b3920bbc-2a92-438d-ab8a-d4e076d3602d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v" Oct 30 21:50:16.202165 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:16.202120 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d83901a8-dbdf-4385-8e0f-af7d2fa13ccf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ccec6d1-gfql7\" (UID: \"d83901a8-dbdf-4385-8e0f-af7d2fa13ccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7" Oct 30 21:50:16.202407 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:16.202218 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6l9n\" (UniqueName: \"kubernetes.io/projected/d83901a8-dbdf-4385-8e0f-af7d2fa13ccf-kube-api-access-z6l9n\") pod \"maas-api-revoke-sub-9ccec6d1-gfql7\" (UID: \"d83901a8-dbdf-4385-8e0f-af7d2fa13ccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7" Oct 30 21:50:16.202407 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:16.202309 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d83901a8-dbdf-4385-8e0f-af7d2fa13ccf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ccec6d1-gfql7\" (UID: \"d83901a8-dbdf-4385-8e0f-af7d2fa13ccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7" Oct 30 21:50:16.202997 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:16.202972 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d83901a8-dbdf-4385-8e0f-af7d2fa13ccf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ccec6d1-gfql7\" (UID: \"d83901a8-dbdf-4385-8e0f-af7d2fa13ccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7" Oct 30 21:50:16.204887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:16.204856 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d83901a8-dbdf-4385-8e0f-af7d2fa13ccf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ccec6d1-gfql7\" (UID: \"d83901a8-dbdf-4385-8e0f-af7d2fa13ccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7" Oct 30 21:50:16.221861 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:16.221822 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6l9n\" (UniqueName: \"kubernetes.io/projected/d83901a8-dbdf-4385-8e0f-af7d2fa13ccf-kube-api-access-z6l9n\") pod \"maas-api-revoke-sub-9ccec6d1-gfql7\" (UID: \"d83901a8-dbdf-4385-8e0f-af7d2fa13ccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7" Oct 30 21:50:16.225469 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:16.225409 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v" Oct 30 21:50:16.253980 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:16.253948 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7" Oct 30 21:50:16.536896 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:16.536863 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v"] Oct 30 21:50:16.537685 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:50:16.537657 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3920bbc_2a92_438d_ab8a_d4e076d3602d.slice/crio-56e9fc67a6fc2bcd5050e6bd30b6469140bea382aecdff307589d3ab2cdb8cb5 WatchSource:0}: Error finding container 56e9fc67a6fc2bcd5050e6bd30b6469140bea382aecdff307589d3ab2cdb8cb5: Status 404 returned error can't find the container with id 56e9fc67a6fc2bcd5050e6bd30b6469140bea382aecdff307589d3ab2cdb8cb5 Oct 30 21:50:16.587450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:16.587415 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7"] Oct 30 21:50:16.588196 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:50:16.587987 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd83901a8_dbdf_4385_8e0f_af7d2fa13ccf.slice/crio-86ed70f81931015186fe4d91d0e18d1c0d7636a3e982d0601a7a1e11193523ad WatchSource:0}: Error finding container 86ed70f81931015186fe4d91d0e18d1c0d7636a3e982d0601a7a1e11193523ad: Status 404 returned error can't find the container with id 86ed70f81931015186fe4d91d0e18d1c0d7636a3e982d0601a7a1e11193523ad Oct 30 21:50:17.228743 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:17.228701 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v" event={"ID":"b3920bbc-2a92-438d-ab8a-d4e076d3602d","Type":"ContainerStarted","Data":"0a5975c8ab3d41333e71ac7cef1acef508b766b18c71d25db29282a4d785a39d"} Oct 30 21:50:17.229279 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:17.228765 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v" event={"ID":"b3920bbc-2a92-438d-ab8a-d4e076d3602d","Type":"ContainerStarted","Data":"56e9fc67a6fc2bcd5050e6bd30b6469140bea382aecdff307589d3ab2cdb8cb5"} Oct 30 21:50:17.230532 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:17.230500 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7" event={"ID":"d83901a8-dbdf-4385-8e0f-af7d2fa13ccf","Type":"ContainerStarted","Data":"6c1de8c136d42d1108109c3b34a24874649c45cc4d04d697add81ce42d91c415"} Oct 30 21:50:17.230635 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:17.230549 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7" event={"ID":"d83901a8-dbdf-4385-8e0f-af7d2fa13ccf","Type":"ContainerStarted","Data":"86ed70f81931015186fe4d91d0e18d1c0d7636a3e982d0601a7a1e11193523ad"} Oct 30 21:50:17.307684 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:17.307626 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v" podStartSLOduration=2.307609014 podStartE2EDuration="2.307609014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:17.26925844 +0000 UTC m=+3004.390332187" watchObservedRunningTime="2026-10-30 21:50:17.307609014 +0000 UTC m=+3004.428682760" Oct 30 21:50:17.308153 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:17.308123 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7" podStartSLOduration=2.308116633 podStartE2EDuration="2.308116633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:17.306273505 +0000 UTC m=+3004.427347252" watchObservedRunningTime="2026-10-30 21:50:17.308116633 +0000 UTC m=+3004.429190379" Oct 30 21:50:18.240597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:18.240559 4926 generic.go:356] "Generic (PLEG): container finished" podID="b3920bbc-2a92-438d-ab8a-d4e076d3602d" containerID="0a5975c8ab3d41333e71ac7cef1acef508b766b18c71d25db29282a4d785a39d" exitCode=0 Oct 30 21:50:18.241146 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:18.240647 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v" event={"ID":"b3920bbc-2a92-438d-ab8a-d4e076d3602d","Type":"ContainerDied","Data":"0a5975c8ab3d41333e71ac7cef1acef508b766b18c71d25db29282a4d785a39d"} Oct 30 21:50:18.242469 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:18.242446 4926 generic.go:356] "Generic (PLEG): container finished" podID="d83901a8-dbdf-4385-8e0f-af7d2fa13ccf" containerID="6c1de8c136d42d1108109c3b34a24874649c45cc4d04d697add81ce42d91c415" exitCode=0 Oct 30 21:50:18.242556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:18.242514 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7" event={"ID":"d83901a8-dbdf-4385-8e0f-af7d2fa13ccf","Type":"ContainerDied","Data":"6c1de8c136d42d1108109c3b34a24874649c45cc4d04d697add81ce42d91c415"} Oct 30 21:50:18.577559 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:18.577468 4926 scope.go:122] "RemoveContainer" containerID="123f6b73e2bfb54f3da5ee58a1d8a38fbe6e996fe902e78093571f612c3bb1eb" Oct 30 21:50:19.479534 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:19.479495 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7" Oct 30 21:50:19.530939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:19.530867 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v" Oct 30 21:50:19.558380 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:19.558341 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3920bbc-2a92-438d-ab8a-d4e076d3602d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3920bbc-2a92-438d-ab8a-d4e076d3602d-maas-api-openshift-ca\") pod \"b3920bbc-2a92-438d-ab8a-d4e076d3602d\" (UID: \"b3920bbc-2a92-438d-ab8a-d4e076d3602d\") " Oct 30 21:50:19.558558 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:19.558390 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b3920bbc-2a92-438d-ab8a-d4e076d3602d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3920bbc-2a92-438d-ab8a-d4e076d3602d-maas-api-xks-ca\") pod \"b3920bbc-2a92-438d-ab8a-d4e076d3602d\" (UID: \"b3920bbc-2a92-438d-ab8a-d4e076d3602d\") " Oct 30 21:50:19.558558 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:19.558455 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3920bbc-2a92-438d-ab8a-d4e076d3602d-kube-api-access-5qnzh\" (UniqueName: \"kubernetes.io/projected/b3920bbc-2a92-438d-ab8a-d4e076d3602d-kube-api-access-5qnzh\") pod \"b3920bbc-2a92-438d-ab8a-d4e076d3602d\" (UID: \"b3920bbc-2a92-438d-ab8a-d4e076d3602d\") " Oct 30 21:50:19.558558 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:19.558505 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d83901a8-dbdf-4385-8e0f-af7d2fa13ccf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d83901a8-dbdf-4385-8e0f-af7d2fa13ccf-maas-api-xks-ca\") pod \"d83901a8-dbdf-4385-8e0f-af7d2fa13ccf\" (UID: \"d83901a8-dbdf-4385-8e0f-af7d2fa13ccf\") " Oct 30 21:50:19.558558 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:19.558551 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d83901a8-dbdf-4385-8e0f-af7d2fa13ccf-kube-api-access-z6l9n\" (UniqueName: \"kubernetes.io/projected/d83901a8-dbdf-4385-8e0f-af7d2fa13ccf-kube-api-access-z6l9n\") pod \"d83901a8-dbdf-4385-8e0f-af7d2fa13ccf\" (UID: \"d83901a8-dbdf-4385-8e0f-af7d2fa13ccf\") " Oct 30 21:50:19.558812 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:19.558659 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d83901a8-dbdf-4385-8e0f-af7d2fa13ccf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d83901a8-dbdf-4385-8e0f-af7d2fa13ccf-maas-api-openshift-ca\") pod \"d83901a8-dbdf-4385-8e0f-af7d2fa13ccf\" (UID: \"d83901a8-dbdf-4385-8e0f-af7d2fa13ccf\") " Oct 30 21:50:19.558812 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:19.558700 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3920bbc-2a92-438d-ab8a-d4e076d3602d-maas-api-openshift-ca" pod "b3920bbc-2a92-438d-ab8a-d4e076d3602d" (UID: "b3920bbc-2a92-438d-ab8a-d4e076d3602d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:19.559137 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:19.559110 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d83901a8-dbdf-4385-8e0f-af7d2fa13ccf-maas-api-openshift-ca" pod "d83901a8-dbdf-4385-8e0f-af7d2fa13ccf" (UID: "d83901a8-dbdf-4385-8e0f-af7d2fa13ccf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:19.559265 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:19.559154 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b3920bbc-2a92-438d-ab8a-d4e076d3602d-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:50:19.561328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:19.561229 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3920bbc-2a92-438d-ab8a-d4e076d3602d-kube-api-access-5qnzh" pod "b3920bbc-2a92-438d-ab8a-d4e076d3602d" (UID: "b3920bbc-2a92-438d-ab8a-d4e076d3602d"). InnerVolumeSpecName "kube-api-access-5qnzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:19.561328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:19.561269 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d83901a8-dbdf-4385-8e0f-af7d2fa13ccf-kube-api-access-z6l9n" pod "d83901a8-dbdf-4385-8e0f-af7d2fa13ccf" (UID: "d83901a8-dbdf-4385-8e0f-af7d2fa13ccf"). InnerVolumeSpecName "kube-api-access-z6l9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:19.561512 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:19.561471 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d83901a8-dbdf-4385-8e0f-af7d2fa13ccf-maas-api-xks-ca" pod "d83901a8-dbdf-4385-8e0f-af7d2fa13ccf" (UID: "d83901a8-dbdf-4385-8e0f-af7d2fa13ccf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:19.562172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:19.562147 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3920bbc-2a92-438d-ab8a-d4e076d3602d-maas-api-xks-ca" pod "b3920bbc-2a92-438d-ab8a-d4e076d3602d" (UID: "b3920bbc-2a92-438d-ab8a-d4e076d3602d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:19.660814 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:19.660777 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b3920bbc-2a92-438d-ab8a-d4e076d3602d-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:50:19.660814 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:19.660808 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qnzh\" (UniqueName: \"kubernetes.io/projected/b3920bbc-2a92-438d-ab8a-d4e076d3602d-kube-api-access-5qnzh\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:50:19.661150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:19.660824 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d83901a8-dbdf-4385-8e0f-af7d2fa13ccf-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:50:19.661150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:19.660838 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6l9n\" (UniqueName: \"kubernetes.io/projected/d83901a8-dbdf-4385-8e0f-af7d2fa13ccf-kube-api-access-z6l9n\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:50:19.661150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:19.660850 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d83901a8-dbdf-4385-8e0f-af7d2fa13ccf-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:50:20.259991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:20.259954 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v" Oct 30 21:50:20.259991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:20.259983 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v" event={"ID":"b3920bbc-2a92-438d-ab8a-d4e076d3602d","Type":"ContainerDied","Data":"56e9fc67a6fc2bcd5050e6bd30b6469140bea382aecdff307589d3ab2cdb8cb5"} Oct 30 21:50:20.260279 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:20.260067 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56e9fc67a6fc2bcd5050e6bd30b6469140bea382aecdff307589d3ab2cdb8cb5" Oct 30 21:50:20.261661 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:20.261636 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7" Oct 30 21:50:20.261791 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:20.261679 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7" event={"ID":"d83901a8-dbdf-4385-8e0f-af7d2fa13ccf","Type":"ContainerDied","Data":"86ed70f81931015186fe4d91d0e18d1c0d7636a3e982d0601a7a1e11193523ad"} Oct 30 21:50:20.261791 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:20.261735 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86ed70f81931015186fe4d91d0e18d1c0d7636a3e982d0601a7a1e11193523ad" Oct 30 21:50:22.283661 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:22.283614 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm" podUID="73ef630c-e7d0-4b76-b194-792276f55a67" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:50:28.360517 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:28.360184 4926 generic.go:356] "Generic (PLEG): container finished" podID="73ef630c-e7d0-4b76-b194-792276f55a67" containerID="571d580e9ca5ccb32eecb779c674cf7269dbd93e9151569fd5643f054e703a02" exitCode=0 Oct 30 21:50:28.360517 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:28.360453 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm" event={"ID":"73ef630c-e7d0-4b76-b194-792276f55a67","Type":"ContainerDied","Data":"571d580e9ca5ccb32eecb779c674cf7269dbd93e9151569fd5643f054e703a02"} Oct 30 21:50:28.491230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:28.491197 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm" Oct 30 21:50:28.612903 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:28.612866 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73ef630c-e7d0-4b76-b194-792276f55a67-kube-api-access-mx6g2\" (UniqueName: \"kubernetes.io/projected/73ef630c-e7d0-4b76-b194-792276f55a67-kube-api-access-mx6g2\") pod \"73ef630c-e7d0-4b76-b194-792276f55a67\" (UID: \"73ef630c-e7d0-4b76-b194-792276f55a67\") " Oct 30 21:50:28.613120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:28.613100 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73ef630c-e7d0-4b76-b194-792276f55a67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73ef630c-e7d0-4b76-b194-792276f55a67-plugins-config-volume\") pod \"73ef630c-e7d0-4b76-b194-792276f55a67\" (UID: \"73ef630c-e7d0-4b76-b194-792276f55a67\") " Oct 30 21:50:28.613506 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:28.613472 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73ef630c-e7d0-4b76-b194-792276f55a67-plugins-config-volume" pod "73ef630c-e7d0-4b76-b194-792276f55a67" (UID: "73ef630c-e7d0-4b76-b194-792276f55a67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:28.613651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:28.613620 4926 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73ef630c-e7d0-4b76-b194-792276f55a67-plugins-config-volume\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:50:28.615206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:28.615178 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73ef630c-e7d0-4b76-b194-792276f55a67-kube-api-access-mx6g2" pod "73ef630c-e7d0-4b76-b194-792276f55a67" (UID: "73ef630c-e7d0-4b76-b194-792276f55a67"). InnerVolumeSpecName "kube-api-access-mx6g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:28.715266 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:28.715179 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx6g2\" (UniqueName: \"kubernetes.io/projected/73ef630c-e7d0-4b76-b194-792276f55a67-kube-api-access-mx6g2\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:50:29.370271 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:29.370182 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm" Oct 30 21:50:29.370271 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:29.370204 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm" event={"ID":"73ef630c-e7d0-4b76-b194-792276f55a67","Type":"ContainerDied","Data":"2d6b41827405e17d49d22051f3d961b27c517cb6a8238c53f881bb2bba527bc1"} Oct 30 21:50:29.370831 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:29.370299 4926 scope.go:122] "RemoveContainer" containerID="571d580e9ca5ccb32eecb779c674cf7269dbd93e9151569fd5643f054e703a02" Oct 30 21:50:29.421058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:29.420988 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm"] Oct 30 21:50:29.432153 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:29.432118 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-2cda44-76f8dccbf4-52gmm"] Oct 30 21:50:31.018573 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:31.018540 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73ef630c-e7d0-4b76-b194-792276f55a67" path="/var/lib/kubelet/pods/73ef630c-e7d0-4b76-b194-792276f55a67/volumes" Oct 30 21:50:36.380491 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.380444 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9"] Oct 30 21:50:36.381222 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.381202 4926 cpu_manager.go:414] "containerMap: removing container" podUID="b3920bbc-2a92-438d-ab8a-d4e076d3602d" containerName="revoke-keys" Oct 30 21:50:36.381222 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.381224 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3920bbc-2a92-438d-ab8a-d4e076d3602d" containerName="revoke-keys" Oct 30 21:50:36.381332 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.381235 4926 cpu_manager.go:414] "containerMap: removing container" podUID="73ef630c-e7d0-4b76-b194-792276f55a67" containerName="payload-processing" Oct 30 21:50:36.381332 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.381241 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73ef630c-e7d0-4b76-b194-792276f55a67" containerName="payload-processing" Oct 30 21:50:36.381332 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.381258 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d83901a8-dbdf-4385-8e0f-af7d2fa13ccf" containerName="revoke-keys" Oct 30 21:50:36.381332 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.381263 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d83901a8-dbdf-4385-8e0f-af7d2fa13ccf" containerName="revoke-keys" Oct 30 21:50:36.381494 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.381389 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3920bbc-2a92-438d-ab8a-d4e076d3602d" containerName="revoke-keys" Oct 30 21:50:36.381494 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.381400 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="73ef630c-e7d0-4b76-b194-792276f55a67" containerName="payload-processing" Oct 30 21:50:36.381494 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.381409 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d83901a8-dbdf-4385-8e0f-af7d2fa13ccf" containerName="revoke-keys" Oct 30 21:50:36.386515 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.386488 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" Oct 30 21:50:36.401715 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.401675 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-22d912-serving-cert\"" Oct 30 21:50:36.403322 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.403292 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9"] Oct 30 21:50:36.526132 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.526081 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963cb97c-107b-440f-8927-10dda8d1628b-metrics-certs\") pod \"maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9\" (UID: \"963cb97c-107b-440f-8927-10dda8d1628b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" Oct 30 21:50:36.526335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.526241 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dwp9\" (UniqueName: \"kubernetes.io/projected/963cb97c-107b-440f-8927-10dda8d1628b-kube-api-access-6dwp9\") pod \"maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9\" (UID: \"963cb97c-107b-440f-8927-10dda8d1628b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" Oct 30 21:50:36.526404 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.526385 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963cb97c-107b-440f-8927-10dda8d1628b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9\" (UID: \"963cb97c-107b-440f-8927-10dda8d1628b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" Oct 30 21:50:36.526472 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.526448 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963cb97c-107b-440f-8927-10dda8d1628b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9\" (UID: \"963cb97c-107b-440f-8927-10dda8d1628b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" Oct 30 21:50:36.627374 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.627328 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dwp9\" (UniqueName: \"kubernetes.io/projected/963cb97c-107b-440f-8927-10dda8d1628b-kube-api-access-6dwp9\") pod \"maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9\" (UID: \"963cb97c-107b-440f-8927-10dda8d1628b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" Oct 30 21:50:36.627593 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.627434 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963cb97c-107b-440f-8927-10dda8d1628b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9\" (UID: \"963cb97c-107b-440f-8927-10dda8d1628b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" Oct 30 21:50:36.627593 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.627497 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963cb97c-107b-440f-8927-10dda8d1628b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9\" (UID: \"963cb97c-107b-440f-8927-10dda8d1628b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" Oct 30 21:50:36.627734 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.627615 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963cb97c-107b-440f-8927-10dda8d1628b-metrics-certs\") pod \"maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9\" (UID: \"963cb97c-107b-440f-8927-10dda8d1628b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" Oct 30 21:50:36.628327 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.628292 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963cb97c-107b-440f-8927-10dda8d1628b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9\" (UID: \"963cb97c-107b-440f-8927-10dda8d1628b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" Oct 30 21:50:36.630649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.630575 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963cb97c-107b-440f-8927-10dda8d1628b-metrics-certs\") pod \"maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9\" (UID: \"963cb97c-107b-440f-8927-10dda8d1628b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" Oct 30 21:50:36.635751 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.635729 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963cb97c-107b-440f-8927-10dda8d1628b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9\" (UID: \"963cb97c-107b-440f-8927-10dda8d1628b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" Oct 30 21:50:36.645979 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.645952 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dwp9\" (UniqueName: \"kubernetes.io/projected/963cb97c-107b-440f-8927-10dda8d1628b-kube-api-access-6dwp9\") pod \"maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9\" (UID: \"963cb97c-107b-440f-8927-10dda8d1628b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" Oct 30 21:50:36.710916 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:36.710877 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" Oct 30 21:50:37.023647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:37.023618 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9"] Oct 30 21:50:37.025079 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:50:37.025044 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963cb97c_107b_440f_8927_10dda8d1628b.slice/crio-ed9f2432f1b397e803aaa82d744285f54a35f2896ab984036bc0d46598ffef65 WatchSource:0}: Error finding container ed9f2432f1b397e803aaa82d744285f54a35f2896ab984036bc0d46598ffef65: Status 404 returned error can't find the container with id ed9f2432f1b397e803aaa82d744285f54a35f2896ab984036bc0d46598ffef65 Oct 30 21:50:37.026548 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:37.026527 4926 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:50:37.455052 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:37.454844 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" event={"ID":"963cb97c-107b-440f-8927-10dda8d1628b","Type":"ContainerStarted","Data":"ed9f2432f1b397e803aaa82d744285f54a35f2896ab984036bc0d46598ffef65"} Oct 30 21:50:37.500057 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:50:37.499967 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:50:38.465592 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:38.465549 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" event={"ID":"963cb97c-107b-440f-8927-10dda8d1628b","Type":"ContainerStarted","Data":"5e743a0b45f6970cfd906055353fcea5908888f1808a432e6a0fa61944e57e40"} Oct 30 21:50:38.466079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:38.465682 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" Oct 30 21:50:38.505619 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:38.505562 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" podStartSLOduration=2.068371433 podStartE2EDuration="2.505546497s" totalImagesPullingTime="437.175064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:38.503808286 +0000 UTC m=+3025.624882032" watchObservedRunningTime="2026-10-30 21:50:38.505546497 +0000 UTC m=+3025.626620237" Oct 30 21:50:44.481222 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:44.481185 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" Oct 30 21:50:50.302849 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.302808 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c"] Oct 30 21:50:50.312704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.309556 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.313421 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.313390 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c"] Oct 30 21:50:50.324654 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.324622 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-e136b8-openshift-default-dockercfg-x42d5\"" Oct 30 21:50:50.435892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.435838 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtbwj\" (UniqueName: \"kubernetes.io/projected/e89edd0d-00f1-4a05-baa6-b053af519b55-kube-api-access-gtbwj\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.436299 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.436275 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-credential-socket\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.436480 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.436462 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e89edd0d-00f1-4a05-baa6-b053af519b55-istiod-ca-cert\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.436628 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.436614 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-envoy\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.436915 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.436896 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-workload-certs\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.437105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.437088 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-token\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.451054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.437327 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-data\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.452485 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.452456 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-podinfo\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.452769 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.452750 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-workload-socket\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.571902 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.571802 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtbwj\" (UniqueName: \"kubernetes.io/projected/e89edd0d-00f1-4a05-baa6-b053af519b55-kube-api-access-gtbwj\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.572204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.572178 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-credential-socket\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.572324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.572309 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e89edd0d-00f1-4a05-baa6-b053af519b55-istiod-ca-cert\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.572426 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.572413 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-envoy\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.572679 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.572656 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-workload-certs\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.572801 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.572786 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-token\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.572992 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.572978 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-data\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.573170 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.573153 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-podinfo\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.573297 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.573284 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-workload-socket\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.574022 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.573997 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-workload-socket\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.576495 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.576470 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-credential-socket\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.578096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.578073 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e89edd0d-00f1-4a05-baa6-b053af519b55-istiod-ca-cert\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.580015 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.578918 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-workload-certs\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.580015 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.579222 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-data\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.595993 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.593708 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-podinfo\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.595993 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.593904 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-envoy\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.610932 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.610900 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtbwj\" (UniqueName: \"kubernetes.io/projected/e89edd0d-00f1-4a05-baa6-b053af519b55-kube-api-access-gtbwj\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.616782 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.616752 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-token\") pod \"e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:50.635435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:50.635405 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:51.005737 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:50:51.005690 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode89edd0d_00f1_4a05_baa6_b053af519b55.slice/crio-e2c1c0ddc995b2584bc53b7c8b6e92852580a8cf1332a2eb1d42cb4ed7b1c7b9 WatchSource:0}: Error finding container e2c1c0ddc995b2584bc53b7c8b6e92852580a8cf1332a2eb1d42cb4ed7b1c7b9: Status 404 returned error can't find the container with id e2c1c0ddc995b2584bc53b7c8b6e92852580a8cf1332a2eb1d42cb4ed7b1c7b9 Oct 30 21:50:51.005885 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:51.005864 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c"] Oct 30 21:50:51.009059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:51.009007 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:50:51.009190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:51.009093 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:50:51.009190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:51.009132 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:50:51.627761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:51.627715 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" event={"ID":"e89edd0d-00f1-4a05-baa6-b053af519b55","Type":"ContainerStarted","Data":"97dbc86ea10608782ca964f1a69998158ded69fbc3ba0cd373748be6457c0284"} Oct 30 21:50:51.628295 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:51.627794 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" event={"ID":"e89edd0d-00f1-4a05-baa6-b053af519b55","Type":"ContainerStarted","Data":"e2c1c0ddc995b2584bc53b7c8b6e92852580a8cf1332a2eb1d42cb4ed7b1c7b9"} Oct 30 21:50:51.635716 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:51.635686 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:51.673264 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:51.673206 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" podStartSLOduration=1.673191098 podStartE2EDuration="1.673191098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:51.668980537 +0000 UTC m=+3038.790054284" watchObservedRunningTime="2026-10-30 21:50:51.673191098 +0000 UTC m=+3038.794264844" Oct 30 21:50:52.642927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:52.642888 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:53.646140 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:53.646095 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:53.647233 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:53.647210 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:50:58.331371 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:50:58.331335 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:51:02.565241 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:02.563128 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl"] Oct 30 21:51:02.565241 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:02.563403 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" podUID="e0dfe0cb-0115-431c-acd8-ac154a18a0ee" containerName="maas-api" containerID="cri-o://cbd9ee9d09878ee206bb1a7f2f55fea1f65f8a098f0bca966d5460cf3ffa7255" gracePeriod=30 Oct 30 21:51:02.730480 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:02.730443 4926 generic.go:356] "Generic (PLEG): container finished" podID="e0dfe0cb-0115-431c-acd8-ac154a18a0ee" containerID="cbd9ee9d09878ee206bb1a7f2f55fea1f65f8a098f0bca966d5460cf3ffa7255" exitCode=0 Oct 30 21:51:02.730709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:02.730559 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" event={"ID":"e0dfe0cb-0115-431c-acd8-ac154a18a0ee","Type":"ContainerDied","Data":"cbd9ee9d09878ee206bb1a7f2f55fea1f65f8a098f0bca966d5460cf3ffa7255"} Oct 30 21:51:02.974097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:02.974071 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" Oct 30 21:51:03.029386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:03.029346 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-kube-api-access-4mccd\" (UniqueName: \"kubernetes.io/projected/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-kube-api-access-4mccd\") pod \"e0dfe0cb-0115-431c-acd8-ac154a18a0ee\" (UID: \"e0dfe0cb-0115-431c-acd8-ac154a18a0ee\") " Oct 30 21:51:03.029589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:03.029474 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-maas-api-tls\") pod \"e0dfe0cb-0115-431c-acd8-ac154a18a0ee\" (UID: \"e0dfe0cb-0115-431c-acd8-ac154a18a0ee\") " Oct 30 21:51:03.029589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:03.029546 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-openshift-service-ca\") pod \"e0dfe0cb-0115-431c-acd8-ac154a18a0ee\" (UID: \"e0dfe0cb-0115-431c-acd8-ac154a18a0ee\") " Oct 30 21:51:03.029707 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:03.029648 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-metrics-certs\") pod \"e0dfe0cb-0115-431c-acd8-ac154a18a0ee\" (UID: \"e0dfe0cb-0115-431c-acd8-ac154a18a0ee\") " Oct 30 21:51:03.031000 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:03.030971 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-openshift-service-ca" pod "e0dfe0cb-0115-431c-acd8-ac154a18a0ee" (UID: "e0dfe0cb-0115-431c-acd8-ac154a18a0ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:03.033750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:03.033718 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-metrics-certs" pod "e0dfe0cb-0115-431c-acd8-ac154a18a0ee" (UID: "e0dfe0cb-0115-431c-acd8-ac154a18a0ee"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:03.033895 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:03.033808 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-kube-api-access-4mccd" pod "e0dfe0cb-0115-431c-acd8-ac154a18a0ee" (UID: "e0dfe0cb-0115-431c-acd8-ac154a18a0ee"). InnerVolumeSpecName "kube-api-access-4mccd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:03.033895 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:03.033812 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-maas-api-tls" pod "e0dfe0cb-0115-431c-acd8-ac154a18a0ee" (UID: "e0dfe0cb-0115-431c-acd8-ac154a18a0ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:03.131370 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:03.131329 4926 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-metrics-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:51:03.131370 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:03.131363 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mccd\" (UniqueName: \"kubernetes.io/projected/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-kube-api-access-4mccd\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:51:03.131370 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:03.131377 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-maas-api-tls\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:51:03.131635 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:03.131390 4926 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0dfe0cb-0115-431c-acd8-ac154a18a0ee-openshift-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:51:03.515767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:03.515723 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz"] Oct 30 21:51:03.688826 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:03.688785 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx"] Oct 30 21:51:03.692719 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:03.692687 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-2cda44-jv9mx"] Oct 30 21:51:03.744461 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:03.744427 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" Oct 30 21:51:03.744461 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:03.744440 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl" event={"ID":"e0dfe0cb-0115-431c-acd8-ac154a18a0ee","Type":"ContainerDied","Data":"55c838f5a74dc9a6d865c38a65d41bd4a17c8c635e87760b5091de2cbaeac038"} Oct 30 21:51:03.744775 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:03.744512 4926 scope.go:122] "RemoveContainer" containerID="cbd9ee9d09878ee206bb1a7f2f55fea1f65f8a098f0bca966d5460cf3ffa7255" Oct 30 21:51:03.827064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:03.826906 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl"] Oct 30 21:51:03.839216 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:03.839178 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-2cda44-78597dddbc-77jpl"] Oct 30 21:51:05.020230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:05.020193 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19fcacf5-242f-4417-9092-45bd2fd7d781" path="/var/lib/kubelet/pods/19fcacf5-242f-4417-9092-45bd2fd7d781/volumes" Oct 30 21:51:05.020916 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:05.020891 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0dfe0cb-0115-431c-acd8-ac154a18a0ee" path="/var/lib/kubelet/pods/e0dfe0cb-0115-431c-acd8-ac154a18a0ee/volumes" Oct 30 21:51:05.448501 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:05.448457 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c"] Oct 30 21:51:05.449109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:05.449089 4926 cpu_manager.go:414] "containerMap: removing container" podUID="e0dfe0cb-0115-431c-acd8-ac154a18a0ee" containerName="maas-api" Oct 30 21:51:05.449172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:05.449109 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0dfe0cb-0115-431c-acd8-ac154a18a0ee" containerName="maas-api" Oct 30 21:51:05.449247 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:05.449238 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0dfe0cb-0115-431c-acd8-ac154a18a0ee" containerName="maas-api" Oct 30 21:51:05.454167 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:05.454151 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c" Oct 30 21:51:05.472361 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:05.472333 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c"] Oct 30 21:51:05.477146 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:05.477116 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:51:05.477288 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:05.477159 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:51:05.587505 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:05.587469 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ce6dcce-38b4-4a80-bfcf-a891f45644eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df9b4e38-8nr8c\" (UID: \"6ce6dcce-38b4-4a80-bfcf-a891f45644eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c" Oct 30 21:51:05.587692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:05.587626 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ce6dcce-38b4-4a80-bfcf-a891f45644eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df9b4e38-8nr8c\" (UID: \"6ce6dcce-38b4-4a80-bfcf-a891f45644eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c" Oct 30 21:51:05.587785 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:05.587751 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7xzb\" (UniqueName: \"kubernetes.io/projected/6ce6dcce-38b4-4a80-bfcf-a891f45644eb-kube-api-access-c7xzb\") pod \"maas-api-revoke-sub-df9b4e38-8nr8c\" (UID: \"6ce6dcce-38b4-4a80-bfcf-a891f45644eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c" Oct 30 21:51:05.689260 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:05.689217 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ce6dcce-38b4-4a80-bfcf-a891f45644eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df9b4e38-8nr8c\" (UID: \"6ce6dcce-38b4-4a80-bfcf-a891f45644eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c" Oct 30 21:51:05.689468 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:05.689378 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7xzb\" (UniqueName: \"kubernetes.io/projected/6ce6dcce-38b4-4a80-bfcf-a891f45644eb-kube-api-access-c7xzb\") pod \"maas-api-revoke-sub-df9b4e38-8nr8c\" (UID: \"6ce6dcce-38b4-4a80-bfcf-a891f45644eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c" Oct 30 21:51:05.689547 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:05.689509 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ce6dcce-38b4-4a80-bfcf-a891f45644eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df9b4e38-8nr8c\" (UID: \"6ce6dcce-38b4-4a80-bfcf-a891f45644eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c" Oct 30 21:51:05.692522 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:05.692491 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ce6dcce-38b4-4a80-bfcf-a891f45644eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df9b4e38-8nr8c\" (UID: \"6ce6dcce-38b4-4a80-bfcf-a891f45644eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c" Oct 30 21:51:05.692689 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:05.692539 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ce6dcce-38b4-4a80-bfcf-a891f45644eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df9b4e38-8nr8c\" (UID: \"6ce6dcce-38b4-4a80-bfcf-a891f45644eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c" Oct 30 21:51:05.712736 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:05.712653 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7xzb\" (UniqueName: \"kubernetes.io/projected/6ce6dcce-38b4-4a80-bfcf-a891f45644eb-kube-api-access-c7xzb\") pod \"maas-api-revoke-sub-df9b4e38-8nr8c\" (UID: \"6ce6dcce-38b4-4a80-bfcf-a891f45644eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c" Oct 30 21:51:05.771575 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:05.771535 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c" Oct 30 21:51:06.111064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:06.110513 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c"] Oct 30 21:51:06.111064 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:51:06.110640 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce6dcce_38b4_4a80_bfcf_a891f45644eb.slice/crio-868088696b319f480665eba96226de2b03ddb1677e0c79d1ed0a9740d92a7bfe WatchSource:0}: Error finding container 868088696b319f480665eba96226de2b03ddb1677e0c79d1ed0a9740d92a7bfe: Status 404 returned error can't find the container with id 868088696b319f480665eba96226de2b03ddb1677e0c79d1ed0a9740d92a7bfe Oct 30 21:51:06.777948 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:06.777849 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c" event={"ID":"6ce6dcce-38b4-4a80-bfcf-a891f45644eb","Type":"ContainerStarted","Data":"d3c7251a96bb42a51d8d58b3612f30c215b6d54e10043315d63ca5d429f8ec4e"} Oct 30 21:51:06.777948 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:06.777917 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c" event={"ID":"6ce6dcce-38b4-4a80-bfcf-a891f45644eb","Type":"ContainerStarted","Data":"868088696b319f480665eba96226de2b03ddb1677e0c79d1ed0a9740d92a7bfe"} Oct 30 21:51:06.816394 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:06.816338 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c" podStartSLOduration=1.816323706 podStartE2EDuration="1.816323706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:06.813429985 +0000 UTC m=+3053.934503732" watchObservedRunningTime="2026-10-30 21:51:06.816323706 +0000 UTC m=+3053.937397451" Oct 30 21:51:07.490112 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:51:07.490075 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:51:07.808053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:07.807905 4926 generic.go:356] "Generic (PLEG): container finished" podID="6ce6dcce-38b4-4a80-bfcf-a891f45644eb" containerID="d3c7251a96bb42a51d8d58b3612f30c215b6d54e10043315d63ca5d429f8ec4e" exitCode=0 Oct 30 21:51:07.808053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:07.807988 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c" event={"ID":"6ce6dcce-38b4-4a80-bfcf-a891f45644eb","Type":"ContainerDied","Data":"d3c7251a96bb42a51d8d58b3612f30c215b6d54e10043315d63ca5d429f8ec4e"} Oct 30 21:51:08.516622 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:08.516546 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz" podUID="856a4b63-41f8-42de-ad43-2a2ffe41dfc4" containerName="payload-processing" containerID="cri-o://535e6631f775189893492b3c472afd72dcdbef93bcf07ea9fb51120ca14a3621" gracePeriod=25 Oct 30 21:51:08.802877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:08.802842 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz" Oct 30 21:51:08.820852 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:08.820820 4926 generic.go:356] "Generic (PLEG): container finished" podID="856a4b63-41f8-42de-ad43-2a2ffe41dfc4" containerID="535e6631f775189893492b3c472afd72dcdbef93bcf07ea9fb51120ca14a3621" exitCode=0 Oct 30 21:51:08.821063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:08.820873 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz" event={"ID":"856a4b63-41f8-42de-ad43-2a2ffe41dfc4","Type":"ContainerDied","Data":"535e6631f775189893492b3c472afd72dcdbef93bcf07ea9fb51120ca14a3621"} Oct 30 21:51:08.821063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:08.820892 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz" Oct 30 21:51:08.821063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:08.820921 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz" event={"ID":"856a4b63-41f8-42de-ad43-2a2ffe41dfc4","Type":"ContainerDied","Data":"01a1f2d4835dd3bef24efc0e0193d4b30f326753a432bfce2d56dfaf3bbde721"} Oct 30 21:51:08.821063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:08.820943 4926 scope.go:122] "RemoveContainer" containerID="535e6631f775189893492b3c472afd72dcdbef93bcf07ea9fb51120ca14a3621" Oct 30 21:51:08.832226 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:08.832198 4926 scope.go:122] "RemoveContainer" containerID="535e6631f775189893492b3c472afd72dcdbef93bcf07ea9fb51120ca14a3621" Oct 30 21:51:08.832558 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:51:08.832528 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"535e6631f775189893492b3c472afd72dcdbef93bcf07ea9fb51120ca14a3621\": container with ID starting with 535e6631f775189893492b3c472afd72dcdbef93bcf07ea9fb51120ca14a3621 not found: ID does not exist" containerID="535e6631f775189893492b3c472afd72dcdbef93bcf07ea9fb51120ca14a3621" Oct 30 21:51:08.832659 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:08.832569 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"535e6631f775189893492b3c472afd72dcdbef93bcf07ea9fb51120ca14a3621"} err="failed to get container status \"535e6631f775189893492b3c472afd72dcdbef93bcf07ea9fb51120ca14a3621\": rpc error: code = NotFound desc = could not find container \"535e6631f775189893492b3c472afd72dcdbef93bcf07ea9fb51120ca14a3621\": container with ID starting with 535e6631f775189893492b3c472afd72dcdbef93bcf07ea9fb51120ca14a3621 not found: ID does not exist" Oct 30 21:51:08.939161 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:08.937500 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/856a4b63-41f8-42de-ad43-2a2ffe41dfc4-kube-api-access-9j5s2\" (UniqueName: \"kubernetes.io/projected/856a4b63-41f8-42de-ad43-2a2ffe41dfc4-kube-api-access-9j5s2\") pod \"856a4b63-41f8-42de-ad43-2a2ffe41dfc4\" (UID: \"856a4b63-41f8-42de-ad43-2a2ffe41dfc4\") " Oct 30 21:51:08.939161 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:08.937662 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/856a4b63-41f8-42de-ad43-2a2ffe41dfc4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/856a4b63-41f8-42de-ad43-2a2ffe41dfc4-plugins-config-volume\") pod \"856a4b63-41f8-42de-ad43-2a2ffe41dfc4\" (UID: \"856a4b63-41f8-42de-ad43-2a2ffe41dfc4\") " Oct 30 21:51:08.939161 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:08.938411 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856a4b63-41f8-42de-ad43-2a2ffe41dfc4-plugins-config-volume" pod "856a4b63-41f8-42de-ad43-2a2ffe41dfc4" (UID: "856a4b63-41f8-42de-ad43-2a2ffe41dfc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:08.942168 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:08.941093 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856a4b63-41f8-42de-ad43-2a2ffe41dfc4-kube-api-access-9j5s2" pod "856a4b63-41f8-42de-ad43-2a2ffe41dfc4" (UID: "856a4b63-41f8-42de-ad43-2a2ffe41dfc4"). InnerVolumeSpecName "kube-api-access-9j5s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:09.013707 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:09.013674 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c" Oct 30 21:51:09.040010 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:09.039972 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j5s2\" (UniqueName: \"kubernetes.io/projected/856a4b63-41f8-42de-ad43-2a2ffe41dfc4-kube-api-access-9j5s2\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:51:09.040010 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:09.040008 4926 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/856a4b63-41f8-42de-ad43-2a2ffe41dfc4-plugins-config-volume\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:51:09.141157 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:09.141124 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ce6dcce-38b4-4a80-bfcf-a891f45644eb-kube-api-access-c7xzb\" (UniqueName: \"kubernetes.io/projected/6ce6dcce-38b4-4a80-bfcf-a891f45644eb-kube-api-access-c7xzb\") pod \"6ce6dcce-38b4-4a80-bfcf-a891f45644eb\" (UID: \"6ce6dcce-38b4-4a80-bfcf-a891f45644eb\") " Oct 30 21:51:09.141360 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:09.141207 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ce6dcce-38b4-4a80-bfcf-a891f45644eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ce6dcce-38b4-4a80-bfcf-a891f45644eb-maas-api-xks-ca\") pod \"6ce6dcce-38b4-4a80-bfcf-a891f45644eb\" (UID: \"6ce6dcce-38b4-4a80-bfcf-a891f45644eb\") " Oct 30 21:51:09.141438 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:09.141419 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ce6dcce-38b4-4a80-bfcf-a891f45644eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ce6dcce-38b4-4a80-bfcf-a891f45644eb-maas-api-openshift-ca\") pod \"6ce6dcce-38b4-4a80-bfcf-a891f45644eb\" (UID: \"6ce6dcce-38b4-4a80-bfcf-a891f45644eb\") " Oct 30 21:51:09.141726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:09.141691 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ce6dcce-38b4-4a80-bfcf-a891f45644eb-maas-api-openshift-ca" pod "6ce6dcce-38b4-4a80-bfcf-a891f45644eb" (UID: "6ce6dcce-38b4-4a80-bfcf-a891f45644eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:09.142170 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:09.142144 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ce6dcce-38b4-4a80-bfcf-a891f45644eb-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:51:09.143394 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:09.143371 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ce6dcce-38b4-4a80-bfcf-a891f45644eb-kube-api-access-c7xzb" pod "6ce6dcce-38b4-4a80-bfcf-a891f45644eb" (UID: "6ce6dcce-38b4-4a80-bfcf-a891f45644eb"). InnerVolumeSpecName "kube-api-access-c7xzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:09.143479 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:09.143409 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ce6dcce-38b4-4a80-bfcf-a891f45644eb-maas-api-xks-ca" pod "6ce6dcce-38b4-4a80-bfcf-a891f45644eb" (UID: "6ce6dcce-38b4-4a80-bfcf-a891f45644eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:09.160834 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:09.160542 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz"] Oct 30 21:51:09.164656 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:09.164631 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9a8ed-55b66f5548-wr2lz"] Oct 30 21:51:09.243538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:09.243494 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7xzb\" (UniqueName: \"kubernetes.io/projected/6ce6dcce-38b4-4a80-bfcf-a891f45644eb-kube-api-access-c7xzb\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:51:09.243538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:09.243533 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ce6dcce-38b4-4a80-bfcf-a891f45644eb-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:51:09.830243 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:09.830156 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c" Oct 30 21:51:09.830243 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:09.830167 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c" event={"ID":"6ce6dcce-38b4-4a80-bfcf-a891f45644eb","Type":"ContainerDied","Data":"868088696b319f480665eba96226de2b03ddb1677e0c79d1ed0a9740d92a7bfe"} Oct 30 21:51:09.830243 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:09.830226 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="868088696b319f480665eba96226de2b03ddb1677e0c79d1ed0a9740d92a7bfe" Oct 30 21:51:11.019614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:11.019580 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="856a4b63-41f8-42de-ad43-2a2ffe41dfc4" path="/var/lib/kubelet/pods/856a4b63-41f8-42de-ad43-2a2ffe41dfc4/volumes" Oct 30 21:51:15.792497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:15.792408 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg"] Oct 30 21:51:15.793354 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:15.793330 4926 cpu_manager.go:414] "containerMap: removing container" podUID="856a4b63-41f8-42de-ad43-2a2ffe41dfc4" containerName="payload-processing" Oct 30 21:51:15.793431 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:15.793356 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856a4b63-41f8-42de-ad43-2a2ffe41dfc4" containerName="payload-processing" Oct 30 21:51:15.793431 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:15.793388 4926 cpu_manager.go:414] "containerMap: removing container" podUID="6ce6dcce-38b4-4a80-bfcf-a891f45644eb" containerName="revoke-keys" Oct 30 21:51:15.793431 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:15.793397 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ce6dcce-38b4-4a80-bfcf-a891f45644eb" containerName="revoke-keys" Oct 30 21:51:15.793605 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:15.793593 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="856a4b63-41f8-42de-ad43-2a2ffe41dfc4" containerName="payload-processing" Oct 30 21:51:15.793645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:15.793611 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ce6dcce-38b4-4a80-bfcf-a891f45644eb" containerName="revoke-keys" Oct 30 21:51:15.797973 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:15.797950 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg" Oct 30 21:51:15.813077 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:15.813019 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg"] Oct 30 21:51:15.818697 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:15.818675 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:51:15.818697 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:15.818672 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:51:15.833508 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:15.833481 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1223d4a3-7b03-487b-a2d4-f9856f89d6a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baf22847-xp5bg\" (UID: \"1223d4a3-7b03-487b-a2d4-f9856f89d6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg" Oct 30 21:51:15.833662 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:15.833544 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1223d4a3-7b03-487b-a2d4-f9856f89d6a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baf22847-xp5bg\" (UID: \"1223d4a3-7b03-487b-a2d4-f9856f89d6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg" Oct 30 21:51:15.833752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:15.833728 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h72f7\" (UniqueName: \"kubernetes.io/projected/1223d4a3-7b03-487b-a2d4-f9856f89d6a7-kube-api-access-h72f7\") pod \"maas-api-revoke-sub-baf22847-xp5bg\" (UID: \"1223d4a3-7b03-487b-a2d4-f9856f89d6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg" Oct 30 21:51:15.938293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:15.935962 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1223d4a3-7b03-487b-a2d4-f9856f89d6a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baf22847-xp5bg\" (UID: \"1223d4a3-7b03-487b-a2d4-f9856f89d6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg" Oct 30 21:51:15.938293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:15.936111 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1223d4a3-7b03-487b-a2d4-f9856f89d6a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baf22847-xp5bg\" (UID: \"1223d4a3-7b03-487b-a2d4-f9856f89d6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg" Oct 30 21:51:15.938293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:15.936242 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h72f7\" (UniqueName: \"kubernetes.io/projected/1223d4a3-7b03-487b-a2d4-f9856f89d6a7-kube-api-access-h72f7\") pod \"maas-api-revoke-sub-baf22847-xp5bg\" (UID: \"1223d4a3-7b03-487b-a2d4-f9856f89d6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg" Oct 30 21:51:15.938293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:15.937541 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1223d4a3-7b03-487b-a2d4-f9856f89d6a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-baf22847-xp5bg\" (UID: \"1223d4a3-7b03-487b-a2d4-f9856f89d6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg" Oct 30 21:51:15.943050 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:15.939933 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1223d4a3-7b03-487b-a2d4-f9856f89d6a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-baf22847-xp5bg\" (UID: \"1223d4a3-7b03-487b-a2d4-f9856f89d6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg" Oct 30 21:51:15.955806 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:15.955775 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h72f7\" (UniqueName: \"kubernetes.io/projected/1223d4a3-7b03-487b-a2d4-f9856f89d6a7-kube-api-access-h72f7\") pod \"maas-api-revoke-sub-baf22847-xp5bg\" (UID: \"1223d4a3-7b03-487b-a2d4-f9856f89d6a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg" Oct 30 21:51:16.114615 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:16.114512 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg" Oct 30 21:51:16.429969 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:16.429934 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg"] Oct 30 21:51:16.435663 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:51:16.435626 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1223d4a3_7b03_487b_a2d4_f9856f89d6a7.slice/crio-c7f459e32595df660af4d09402d2e9574d87eefcbc808d0905850d47fa5e5a9a WatchSource:0}: Error finding container c7f459e32595df660af4d09402d2e9574d87eefcbc808d0905850d47fa5e5a9a: Status 404 returned error can't find the container with id c7f459e32595df660af4d09402d2e9574d87eefcbc808d0905850d47fa5e5a9a Oct 30 21:51:16.897782 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:16.897731 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg" event={"ID":"1223d4a3-7b03-487b-a2d4-f9856f89d6a7","Type":"ContainerStarted","Data":"3355b38126e4007fe95ac0368957e0a2071848a3e81514c7965bcb13bf1dd9a2"} Oct 30 21:51:16.898275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:16.897814 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg" event={"ID":"1223d4a3-7b03-487b-a2d4-f9856f89d6a7","Type":"ContainerStarted","Data":"c7f459e32595df660af4d09402d2e9574d87eefcbc808d0905850d47fa5e5a9a"} Oct 30 21:51:16.936751 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:16.936629 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg" podStartSLOduration=1.936613068 podStartE2EDuration="1.936613068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:16.933250095 +0000 UTC m=+3064.054323841" watchObservedRunningTime="2026-10-30 21:51:16.936613068 +0000 UTC m=+3064.057686814" Oct 30 21:51:17.909796 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:17.909697 4926 generic.go:356] "Generic (PLEG): container finished" podID="1223d4a3-7b03-487b-a2d4-f9856f89d6a7" containerID="3355b38126e4007fe95ac0368957e0a2071848a3e81514c7965bcb13bf1dd9a2" exitCode=0 Oct 30 21:51:17.910219 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:17.909785 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg" event={"ID":"1223d4a3-7b03-487b-a2d4-f9856f89d6a7","Type":"ContainerDied","Data":"3355b38126e4007fe95ac0368957e0a2071848a3e81514c7965bcb13bf1dd9a2"} Oct 30 21:51:18.621936 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:18.621900 4926 scope.go:122] "RemoveContainer" containerID="d55b08ce2a5ee76f879c413f3076b1a5f1a5171785f2036c3dc6f6c17a628219" Oct 30 21:51:18.633447 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:18.633420 4926 scope.go:122] "RemoveContainer" containerID="50743f30bb624e802d80c628d7ee14261cf97d693943ec95f9d2d1baeea115b3" Oct 30 21:51:19.088163 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:19.088132 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg" Oct 30 21:51:19.185547 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:19.185456 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1223d4a3-7b03-487b-a2d4-f9856f89d6a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1223d4a3-7b03-487b-a2d4-f9856f89d6a7-maas-api-openshift-ca\") pod \"1223d4a3-7b03-487b-a2d4-f9856f89d6a7\" (UID: \"1223d4a3-7b03-487b-a2d4-f9856f89d6a7\") " Oct 30 21:51:19.185770 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:19.185752 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1223d4a3-7b03-487b-a2d4-f9856f89d6a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1223d4a3-7b03-487b-a2d4-f9856f89d6a7-maas-api-xks-ca\") pod \"1223d4a3-7b03-487b-a2d4-f9856f89d6a7\" (UID: \"1223d4a3-7b03-487b-a2d4-f9856f89d6a7\") " Oct 30 21:51:19.185820 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:19.185791 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1223d4a3-7b03-487b-a2d4-f9856f89d6a7-kube-api-access-h72f7\" (UniqueName: \"kubernetes.io/projected/1223d4a3-7b03-487b-a2d4-f9856f89d6a7-kube-api-access-h72f7\") pod \"1223d4a3-7b03-487b-a2d4-f9856f89d6a7\" (UID: \"1223d4a3-7b03-487b-a2d4-f9856f89d6a7\") " Oct 30 21:51:19.185929 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:19.185901 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1223d4a3-7b03-487b-a2d4-f9856f89d6a7-maas-api-openshift-ca" pod "1223d4a3-7b03-487b-a2d4-f9856f89d6a7" (UID: "1223d4a3-7b03-487b-a2d4-f9856f89d6a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:19.186376 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:19.186348 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1223d4a3-7b03-487b-a2d4-f9856f89d6a7-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:51:19.188126 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:19.188095 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1223d4a3-7b03-487b-a2d4-f9856f89d6a7-kube-api-access-h72f7" pod "1223d4a3-7b03-487b-a2d4-f9856f89d6a7" (UID: "1223d4a3-7b03-487b-a2d4-f9856f89d6a7"). InnerVolumeSpecName "kube-api-access-h72f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:19.188256 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:19.188223 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1223d4a3-7b03-487b-a2d4-f9856f89d6a7-maas-api-xks-ca" pod "1223d4a3-7b03-487b-a2d4-f9856f89d6a7" (UID: "1223d4a3-7b03-487b-a2d4-f9856f89d6a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:19.287706 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:19.287668 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1223d4a3-7b03-487b-a2d4-f9856f89d6a7-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:51:19.287706 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:19.287703 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h72f7\" (UniqueName: \"kubernetes.io/projected/1223d4a3-7b03-487b-a2d4-f9856f89d6a7-kube-api-access-h72f7\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:51:19.935198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:19.935162 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg" Oct 30 21:51:19.935198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:19.935191 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg" event={"ID":"1223d4a3-7b03-487b-a2d4-f9856f89d6a7","Type":"ContainerDied","Data":"c7f459e32595df660af4d09402d2e9574d87eefcbc808d0905850d47fa5e5a9a"} Oct 30 21:51:19.935481 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:19.935245 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7f459e32595df660af4d09402d2e9574d87eefcbc808d0905850d47fa5e5a9a" Oct 30 21:51:26.692516 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:26.692476 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv"] Oct 30 21:51:26.693112 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:26.693095 4926 cpu_manager.go:414] "containerMap: removing container" podUID="1223d4a3-7b03-487b-a2d4-f9856f89d6a7" containerName="revoke-keys" Oct 30 21:51:26.693186 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:26.693112 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1223d4a3-7b03-487b-a2d4-f9856f89d6a7" containerName="revoke-keys" Oct 30 21:51:26.693227 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:26.693217 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="1223d4a3-7b03-487b-a2d4-f9856f89d6a7" containerName="revoke-keys" Oct 30 21:51:26.698069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:26.698025 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv" Oct 30 21:51:26.714139 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:26.714101 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv"] Oct 30 21:51:26.719507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:26.718860 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:51:26.719507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:26.718860 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:51:26.793989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:26.793951 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6768e51-6568-4cd7-a64d-0ae77e339ce0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f80b7bf0-zrfhv\" (UID: \"a6768e51-6568-4cd7-a64d-0ae77e339ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv" Oct 30 21:51:26.794269 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:26.794205 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqj6m\" (UniqueName: \"kubernetes.io/projected/a6768e51-6568-4cd7-a64d-0ae77e339ce0-kube-api-access-bqj6m\") pod \"maas-api-revoke-sub-f80b7bf0-zrfhv\" (UID: \"a6768e51-6568-4cd7-a64d-0ae77e339ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv" Oct 30 21:51:26.794350 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:26.794280 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6768e51-6568-4cd7-a64d-0ae77e339ce0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f80b7bf0-zrfhv\" (UID: \"a6768e51-6568-4cd7-a64d-0ae77e339ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv" Oct 30 21:51:26.895347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:26.895298 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6768e51-6568-4cd7-a64d-0ae77e339ce0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f80b7bf0-zrfhv\" (UID: \"a6768e51-6568-4cd7-a64d-0ae77e339ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv" Oct 30 21:51:26.895584 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:26.895395 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6768e51-6568-4cd7-a64d-0ae77e339ce0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f80b7bf0-zrfhv\" (UID: \"a6768e51-6568-4cd7-a64d-0ae77e339ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv" Oct 30 21:51:26.895662 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:26.895600 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqj6m\" (UniqueName: \"kubernetes.io/projected/a6768e51-6568-4cd7-a64d-0ae77e339ce0-kube-api-access-bqj6m\") pod \"maas-api-revoke-sub-f80b7bf0-zrfhv\" (UID: \"a6768e51-6568-4cd7-a64d-0ae77e339ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv" Oct 30 21:51:26.896292 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:26.896262 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6768e51-6568-4cd7-a64d-0ae77e339ce0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f80b7bf0-zrfhv\" (UID: \"a6768e51-6568-4cd7-a64d-0ae77e339ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv" Oct 30 21:51:26.898483 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:26.898456 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6768e51-6568-4cd7-a64d-0ae77e339ce0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f80b7bf0-zrfhv\" (UID: \"a6768e51-6568-4cd7-a64d-0ae77e339ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv" Oct 30 21:51:26.915057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:26.915005 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqj6m\" (UniqueName: \"kubernetes.io/projected/a6768e51-6568-4cd7-a64d-0ae77e339ce0-kube-api-access-bqj6m\") pod \"maas-api-revoke-sub-f80b7bf0-zrfhv\" (UID: \"a6768e51-6568-4cd7-a64d-0ae77e339ce0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv" Oct 30 21:51:27.015767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:27.015684 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv" Oct 30 21:51:27.336181 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:27.336144 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv"] Oct 30 21:51:27.337398 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:51:27.337366 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6768e51_6568_4cd7_a64d_0ae77e339ce0.slice/crio-bb19eb8e6fa0ba659dd50d86d874fe6abb9fdff20fd4105041cb18c9a6045595 WatchSource:0}: Error finding container bb19eb8e6fa0ba659dd50d86d874fe6abb9fdff20fd4105041cb18c9a6045595: Status 404 returned error can't find the container with id bb19eb8e6fa0ba659dd50d86d874fe6abb9fdff20fd4105041cb18c9a6045595 Oct 30 21:51:28.006418 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:28.006318 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv" event={"ID":"a6768e51-6568-4cd7-a64d-0ae77e339ce0","Type":"ContainerStarted","Data":"d88dec00f9a0ce7ee358e97497567712ea5016439789de87f5b7974128cd8b92"} Oct 30 21:51:28.006418 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:28.006380 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv" event={"ID":"a6768e51-6568-4cd7-a64d-0ae77e339ce0","Type":"ContainerStarted","Data":"bb19eb8e6fa0ba659dd50d86d874fe6abb9fdff20fd4105041cb18c9a6045595"} Oct 30 21:51:28.044506 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:28.044442 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv" podStartSLOduration=2.044427404 podStartE2EDuration="2.044427404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:28.041165941 +0000 UTC m=+3075.162239689" watchObservedRunningTime="2026-10-30 21:51:28.044427404 +0000 UTC m=+3075.165501150" Oct 30 21:51:29.019805 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:29.019764 4926 generic.go:356] "Generic (PLEG): container finished" podID="a6768e51-6568-4cd7-a64d-0ae77e339ce0" containerID="d88dec00f9a0ce7ee358e97497567712ea5016439789de87f5b7974128cd8b92" exitCode=0 Oct 30 21:51:29.020272 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:29.019853 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv" event={"ID":"a6768e51-6568-4cd7-a64d-0ae77e339ce0","Type":"ContainerDied","Data":"d88dec00f9a0ce7ee358e97497567712ea5016439789de87f5b7974128cd8b92"} Oct 30 21:51:30.190422 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:30.190395 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv" Oct 30 21:51:30.340724 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:30.340623 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6768e51-6568-4cd7-a64d-0ae77e339ce0-kube-api-access-bqj6m\" (UniqueName: \"kubernetes.io/projected/a6768e51-6568-4cd7-a64d-0ae77e339ce0-kube-api-access-bqj6m\") pod \"a6768e51-6568-4cd7-a64d-0ae77e339ce0\" (UID: \"a6768e51-6568-4cd7-a64d-0ae77e339ce0\") " Oct 30 21:51:30.340913 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:30.340810 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6768e51-6568-4cd7-a64d-0ae77e339ce0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6768e51-6568-4cd7-a64d-0ae77e339ce0-maas-api-xks-ca\") pod \"a6768e51-6568-4cd7-a64d-0ae77e339ce0\" (UID: \"a6768e51-6568-4cd7-a64d-0ae77e339ce0\") " Oct 30 21:51:30.340913 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:30.340848 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6768e51-6568-4cd7-a64d-0ae77e339ce0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6768e51-6568-4cd7-a64d-0ae77e339ce0-maas-api-openshift-ca\") pod \"a6768e51-6568-4cd7-a64d-0ae77e339ce0\" (UID: \"a6768e51-6568-4cd7-a64d-0ae77e339ce0\") " Oct 30 21:51:30.341363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:30.341327 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6768e51-6568-4cd7-a64d-0ae77e339ce0-maas-api-openshift-ca" pod "a6768e51-6568-4cd7-a64d-0ae77e339ce0" (UID: "a6768e51-6568-4cd7-a64d-0ae77e339ce0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:30.342987 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:30.342955 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6768e51-6568-4cd7-a64d-0ae77e339ce0-kube-api-access-bqj6m" pod "a6768e51-6568-4cd7-a64d-0ae77e339ce0" (UID: "a6768e51-6568-4cd7-a64d-0ae77e339ce0"). InnerVolumeSpecName "kube-api-access-bqj6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:30.343146 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:30.343010 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6768e51-6568-4cd7-a64d-0ae77e339ce0-maas-api-xks-ca" pod "a6768e51-6568-4cd7-a64d-0ae77e339ce0" (UID: "a6768e51-6568-4cd7-a64d-0ae77e339ce0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:30.442533 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:30.442495 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a6768e51-6568-4cd7-a64d-0ae77e339ce0-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:51:30.442533 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:30.442530 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a6768e51-6568-4cd7-a64d-0ae77e339ce0-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:51:30.442533 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:30.442541 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqj6m\" (UniqueName: \"kubernetes.io/projected/a6768e51-6568-4cd7-a64d-0ae77e339ce0-kube-api-access-bqj6m\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:51:31.037656 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:31.037619 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv" Oct 30 21:51:31.037656 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:31.037641 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv" event={"ID":"a6768e51-6568-4cd7-a64d-0ae77e339ce0","Type":"ContainerDied","Data":"bb19eb8e6fa0ba659dd50d86d874fe6abb9fdff20fd4105041cb18c9a6045595"} Oct 30 21:51:31.037884 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:31.037698 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb19eb8e6fa0ba659dd50d86d874fe6abb9fdff20fd4105041cb18c9a6045595" Oct 30 21:51:37.088908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.088861 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f"] Oct 30 21:51:37.089801 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.089772 4926 cpu_manager.go:414] "containerMap: removing container" podUID="a6768e51-6568-4cd7-a64d-0ae77e339ce0" containerName="revoke-keys" Oct 30 21:51:37.089801 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.089802 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6768e51-6568-4cd7-a64d-0ae77e339ce0" containerName="revoke-keys" Oct 30 21:51:37.090009 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.089995 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6768e51-6568-4cd7-a64d-0ae77e339ce0" containerName="revoke-keys" Oct 30 21:51:37.094509 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.094486 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" Oct 30 21:51:37.111144 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.111113 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f"] Oct 30 21:51:37.115853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.115814 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-e136b8-serving-cert\"" Oct 30 21:51:37.129910 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.129868 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctmqf\" (UniqueName: \"kubernetes.io/projected/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-kube-api-access-ctmqf\") pod \"maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f\" (UID: \"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" Oct 30 21:51:37.130102 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.130065 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-metrics-certs\") pod \"maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f\" (UID: \"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" Oct 30 21:51:37.130172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.130135 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f\" (UID: \"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" Oct 30 21:51:37.130247 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.130229 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f\" (UID: \"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" Oct 30 21:51:37.174779 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.174744 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e3e153-gs7p4"] Oct 30 21:51:37.179005 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.178986 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e3e153-gs7p4" Oct 30 21:51:37.198768 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.198735 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e3e153-gs7p4"] Oct 30 21:51:37.199872 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.199823 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:51:37.200104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.200080 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:51:37.231416 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.231375 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-metrics-certs\") pod \"maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f\" (UID: \"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" Oct 30 21:51:37.231416 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.231417 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3285ca61-cc2b-4c14-9113-e2aaafcc7e53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7e3e153-gs7p4\" (UID: \"3285ca61-cc2b-4c14-9113-e2aaafcc7e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e3e153-gs7p4" Oct 30 21:51:37.231665 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.231464 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f\" (UID: \"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" Oct 30 21:51:37.231665 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.231500 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f\" (UID: \"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" Oct 30 21:51:37.231665 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.231630 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95vvz\" (UniqueName: \"kubernetes.io/projected/3285ca61-cc2b-4c14-9113-e2aaafcc7e53-kube-api-access-95vvz\") pod \"maas-api-revoke-sub-a7e3e153-gs7p4\" (UID: \"3285ca61-cc2b-4c14-9113-e2aaafcc7e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e3e153-gs7p4" Oct 30 21:51:37.231840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.231754 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctmqf\" (UniqueName: \"kubernetes.io/projected/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-kube-api-access-ctmqf\") pod \"maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f\" (UID: \"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" Oct 30 21:51:37.231840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.231781 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3285ca61-cc2b-4c14-9113-e2aaafcc7e53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7e3e153-gs7p4\" (UID: \"3285ca61-cc2b-4c14-9113-e2aaafcc7e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e3e153-gs7p4" Oct 30 21:51:37.232466 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.232431 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f\" (UID: \"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" Oct 30 21:51:37.234304 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.234280 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f\" (UID: \"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" Oct 30 21:51:37.234405 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.234385 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-metrics-certs\") pod \"maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f\" (UID: \"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" Oct 30 21:51:37.250883 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.250851 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctmqf\" (UniqueName: \"kubernetes.io/projected/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-kube-api-access-ctmqf\") pod \"maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f\" (UID: \"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" Oct 30 21:51:37.333550 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.333506 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3285ca61-cc2b-4c14-9113-e2aaafcc7e53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7e3e153-gs7p4\" (UID: \"3285ca61-cc2b-4c14-9113-e2aaafcc7e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e3e153-gs7p4" Oct 30 21:51:37.334532 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.334500 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3285ca61-cc2b-4c14-9113-e2aaafcc7e53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7e3e153-gs7p4\" (UID: \"3285ca61-cc2b-4c14-9113-e2aaafcc7e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e3e153-gs7p4" Oct 30 21:51:37.337065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.337014 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3285ca61-cc2b-4c14-9113-e2aaafcc7e53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7e3e153-gs7p4\" (UID: \"3285ca61-cc2b-4c14-9113-e2aaafcc7e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e3e153-gs7p4" Oct 30 21:51:37.338746 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.338509 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3285ca61-cc2b-4c14-9113-e2aaafcc7e53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7e3e153-gs7p4\" (UID: \"3285ca61-cc2b-4c14-9113-e2aaafcc7e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e3e153-gs7p4" Oct 30 21:51:37.338862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.338804 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95vvz\" (UniqueName: \"kubernetes.io/projected/3285ca61-cc2b-4c14-9113-e2aaafcc7e53-kube-api-access-95vvz\") pod \"maas-api-revoke-sub-a7e3e153-gs7p4\" (UID: \"3285ca61-cc2b-4c14-9113-e2aaafcc7e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e3e153-gs7p4" Oct 30 21:51:37.364492 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.363481 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95vvz\" (UniqueName: \"kubernetes.io/projected/3285ca61-cc2b-4c14-9113-e2aaafcc7e53-kube-api-access-95vvz\") pod \"maas-api-revoke-sub-a7e3e153-gs7p4\" (UID: \"3285ca61-cc2b-4c14-9113-e2aaafcc7e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e3e153-gs7p4" Oct 30 21:51:37.412984 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.412935 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" Oct 30 21:51:37.500053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.499826 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e3e153-gs7p4" Oct 30 21:51:37.511750 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:51:37.511240 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:51:37.723837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.723801 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f"] Oct 30 21:51:37.724321 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:51:37.724294 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89ebf4c_90ed_41d6_b85d_e59d44e3ebbc.slice/crio-6fa370107bd559aef85c18ab2790c6c007bb8c47c683173e581a00b841091392 WatchSource:0}: Error finding container 6fa370107bd559aef85c18ab2790c6c007bb8c47c683173e581a00b841091392: Status 404 returned error can't find the container with id 6fa370107bd559aef85c18ab2790c6c007bb8c47c683173e581a00b841091392 Oct 30 21:51:37.839656 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:37.839619 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e3e153-gs7p4"] Oct 30 21:51:37.843341 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:51:37.843299 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3285ca61_cc2b_4c14_9113_e2aaafcc7e53.slice/crio-0cd9f775d8762e096d723a4121850d9cfa8ba8f9f725ce51b1e5892cb5f9d492 WatchSource:0}: Error finding container 0cd9f775d8762e096d723a4121850d9cfa8ba8f9f725ce51b1e5892cb5f9d492: Status 404 returned error can't find the container with id 0cd9f775d8762e096d723a4121850d9cfa8ba8f9f725ce51b1e5892cb5f9d492 Oct 30 21:51:38.106215 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:38.105838 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" event={"ID":"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc","Type":"ContainerStarted","Data":"6fa370107bd559aef85c18ab2790c6c007bb8c47c683173e581a00b841091392"} Oct 30 21:51:38.107634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:38.107602 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e3e153-gs7p4" event={"ID":"3285ca61-cc2b-4c14-9113-e2aaafcc7e53","Type":"ContainerStarted","Data":"0cd9f775d8762e096d723a4121850d9cfa8ba8f9f725ce51b1e5892cb5f9d492"} Oct 30 21:51:39.118733 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:39.118634 4926 generic.go:356] "Generic (PLEG): container finished" podID="3285ca61-cc2b-4c14-9113-e2aaafcc7e53" containerID="1d3921eb970d966cb7d5b7788ca9ffc9ca469d26f7ee4d1906e2a3c7bd9b17f1" exitCode=0 Oct 30 21:51:39.119141 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:39.118718 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e3e153-gs7p4" event={"ID":"3285ca61-cc2b-4c14-9113-e2aaafcc7e53","Type":"ContainerDied","Data":"1d3921eb970d966cb7d5b7788ca9ffc9ca469d26f7ee4d1906e2a3c7bd9b17f1"} Oct 30 21:51:39.120640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:39.120610 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" event={"ID":"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc","Type":"ContainerStarted","Data":"1ba31048818e8e42a460953e5539d7ca1f1fcc48963e7011b949614f6f2bd63b"} Oct 30 21:51:39.120819 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:39.120690 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" Oct 30 21:51:39.200305 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:39.200238 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" podStartSLOduration=1.883543325 podStartE2EDuration="2.200219344s" totalImagesPullingTime="316.676019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:39.197396644 +0000 UTC m=+3086.318470393" watchObservedRunningTime="2026-10-30 21:51:39.200219344 +0000 UTC m=+3086.321293094" Oct 30 21:51:40.030108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:40.030072 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91171a3c-v5hdk"] Oct 30 21:51:40.041004 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:40.040968 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91171a3c-v5hdk"] Oct 30 21:51:40.301329 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:40.300863 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e3e153-gs7p4" Oct 30 21:51:40.388723 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:40.388678 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3285ca61-cc2b-4c14-9113-e2aaafcc7e53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3285ca61-cc2b-4c14-9113-e2aaafcc7e53-maas-api-openshift-ca\") pod \"3285ca61-cc2b-4c14-9113-e2aaafcc7e53\" (UID: \"3285ca61-cc2b-4c14-9113-e2aaafcc7e53\") " Oct 30 21:51:40.388928 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:40.388791 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3285ca61-cc2b-4c14-9113-e2aaafcc7e53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3285ca61-cc2b-4c14-9113-e2aaafcc7e53-maas-api-xks-ca\") pod \"3285ca61-cc2b-4c14-9113-e2aaafcc7e53\" (UID: \"3285ca61-cc2b-4c14-9113-e2aaafcc7e53\") " Oct 30 21:51:40.388928 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:40.388835 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3285ca61-cc2b-4c14-9113-e2aaafcc7e53-kube-api-access-95vvz\" (UniqueName: \"kubernetes.io/projected/3285ca61-cc2b-4c14-9113-e2aaafcc7e53-kube-api-access-95vvz\") pod \"3285ca61-cc2b-4c14-9113-e2aaafcc7e53\" (UID: \"3285ca61-cc2b-4c14-9113-e2aaafcc7e53\") " Oct 30 21:51:40.389146 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:40.389117 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3285ca61-cc2b-4c14-9113-e2aaafcc7e53-maas-api-openshift-ca" pod "3285ca61-cc2b-4c14-9113-e2aaafcc7e53" (UID: "3285ca61-cc2b-4c14-9113-e2aaafcc7e53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:51:40.389566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:40.389538 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3285ca61-cc2b-4c14-9113-e2aaafcc7e53-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:51:40.392400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:40.392371 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3285ca61-cc2b-4c14-9113-e2aaafcc7e53-maas-api-xks-ca" pod "3285ca61-cc2b-4c14-9113-e2aaafcc7e53" (UID: "3285ca61-cc2b-4c14-9113-e2aaafcc7e53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:51:40.392543 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:40.392401 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3285ca61-cc2b-4c14-9113-e2aaafcc7e53-kube-api-access-95vvz" pod "3285ca61-cc2b-4c14-9113-e2aaafcc7e53" (UID: "3285ca61-cc2b-4c14-9113-e2aaafcc7e53"). InnerVolumeSpecName "kube-api-access-95vvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:51:40.491433 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:40.491390 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3285ca61-cc2b-4c14-9113-e2aaafcc7e53-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:51:40.491433 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:40.491428 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95vvz\" (UniqueName: \"kubernetes.io/projected/3285ca61-cc2b-4c14-9113-e2aaafcc7e53-kube-api-access-95vvz\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:51:41.020049 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:41.019990 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d131007a-17b7-4e82-9e38-96b96aed4932" path="/var/lib/kubelet/pods/d131007a-17b7-4e82-9e38-96b96aed4932/volumes" Oct 30 21:51:41.138274 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:41.138241 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e3e153-gs7p4" Oct 30 21:51:41.138452 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:41.138280 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7e3e153-gs7p4" event={"ID":"3285ca61-cc2b-4c14-9113-e2aaafcc7e53","Type":"ContainerDied","Data":"0cd9f775d8762e096d723a4121850d9cfa8ba8f9f725ce51b1e5892cb5f9d492"} Oct 30 21:51:41.138452 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:41.138336 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cd9f775d8762e096d723a4121850d9cfa8ba8f9f725ce51b1e5892cb5f9d492" Oct 30 21:51:45.134708 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:45.134676 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" Oct 30 21:51:45.503301 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:45.503257 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:51:48.337487 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:48.337445 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl"] Oct 30 21:51:51.028703 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:51.028658 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr"] Oct 30 21:51:51.040208 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:51.040173 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-276c4241-xsscr"] Oct 30 21:51:52.027717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:52.027683 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc"] Oct 30 21:51:52.038667 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:52.038635 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46946180-xfqtc"] Oct 30 21:51:53.019889 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:53.019851 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f2d8855-6268-45d1-ba54-b27bb7913608" path="/var/lib/kubelet/pods/5f2d8855-6268-45d1-ba54-b27bb7913608/volumes" Oct 30 21:51:53.020324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:53.020308 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f77ba38-3140-4f05-b127-ea27b1967882" path="/var/lib/kubelet/pods/9f77ba38-3140-4f05-b127-ea27b1967882/volumes" Oct 30 21:51:57.347090 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:57.346984 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv"] Oct 30 21:51:58.480144 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.480089 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45"] Oct 30 21:51:58.481796 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.481758 4926 cpu_manager.go:414] "containerMap: removing container" podUID="3285ca61-cc2b-4c14-9113-e2aaafcc7e53" containerName="revoke-keys" Oct 30 21:51:58.481796 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.481795 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3285ca61-cc2b-4c14-9113-e2aaafcc7e53" containerName="revoke-keys" Oct 30 21:51:58.482114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.482091 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="3285ca61-cc2b-4c14-9113-e2aaafcc7e53" containerName="revoke-keys" Oct 30 21:51:58.488564 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.488540 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.505557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.505513 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-be3ada-openshift-default-dockercfg-d7lq5\"" Oct 30 21:51:58.521012 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.520978 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45"] Oct 30 21:51:58.562697 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.562659 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-workload-certs\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.562892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.562722 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istiod-ca-cert\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.562892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.562848 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-data\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.563078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.562898 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-podinfo\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.563078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.562977 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-workload-socket\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.563078 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.563054 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-token\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.563206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.563144 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-credential-socket\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.563256 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.563212 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-envoy\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.563307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.563287 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdpqt\" (UniqueName: \"kubernetes.io/projected/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-kube-api-access-mdpqt\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.676054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.669381 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-podinfo\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.676054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.669461 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-workload-socket\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.676054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.669542 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-token\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.676054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.669584 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-credential-socket\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.676054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.669653 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-envoy\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.676054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.669727 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdpqt\" (UniqueName: \"kubernetes.io/projected/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-kube-api-access-mdpqt\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.676054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.669808 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-workload-certs\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.676054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.669847 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istiod-ca-cert\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.676054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.669936 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-data\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.676054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.674503 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-credential-socket\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.686061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.681388 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-workload-certs\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.686061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.682079 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istiod-ca-cert\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.686061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.682365 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-workload-socket\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.686061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.682831 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-data\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.720059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.708322 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-podinfo\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.729807 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.729730 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdpqt\" (UniqueName: \"kubernetes.io/projected/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-kube-api-access-mdpqt\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.746247 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.746100 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-token\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.753110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.749401 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-envoy\") pod \"e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:58.820589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:58.820190 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:51:59.156387 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:59.156350 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45"] Oct 30 21:51:59.156548 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:51:59.156493 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ff8991_1ebf_4cf8_96c5_bbc0fd96d6a6.slice/crio-3d3368b091b41345e306ac328c1e459b79a91d0122903af16cd9840f813962bf WatchSource:0}: Error finding container 3d3368b091b41345e306ac328c1e459b79a91d0122903af16cd9840f813962bf: Status 404 returned error can't find the container with id 3d3368b091b41345e306ac328c1e459b79a91d0122903af16cd9840f813962bf Oct 30 21:51:59.159771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:59.159729 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:51:59.159874 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:59.159827 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:51:59.159916 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:59.159870 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:51:59.302052 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:59.302001 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" event={"ID":"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6","Type":"ContainerStarted","Data":"2068931c23566279d1cb1eb15b62341ab40c38c9d3d0c4d36f7aefd8643d0b06"} Oct 30 21:51:59.302273 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:59.302116 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" event={"ID":"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6","Type":"ContainerStarted","Data":"3d3368b091b41345e306ac328c1e459b79a91d0122903af16cd9840f813962bf"} Oct 30 21:51:59.349713 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:59.349655 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" podStartSLOduration=1.349638989 podStartE2EDuration="1.349638989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:59.345879167 +0000 UTC m=+3106.466952910" watchObservedRunningTime="2026-10-30 21:51:59.349638989 +0000 UTC m=+3106.470712734" Oct 30 21:51:59.823292 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:51:59.823248 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:52:00.829571 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:00.829537 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:52:01.323057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:01.322985 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:52:01.324163 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:01.324135 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:52:01.956098 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:01.956062 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn"] Oct 30 21:52:01.960546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:01.960519 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn" Oct 30 21:52:01.975495 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:01.975460 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:52:01.975655 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:01.975530 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:52:01.982628 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:01.982588 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn"] Oct 30 21:52:02.027739 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:02.027689 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d586fea-871e-4f69-bb4b-80d4a8c24a1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn\" (UID: \"8d586fea-871e-4f69-bb4b-80d4a8c24a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn" Oct 30 21:52:02.028028 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:02.028007 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h4lc\" (UniqueName: \"kubernetes.io/projected/8d586fea-871e-4f69-bb4b-80d4a8c24a1f-kube-api-access-6h4lc\") pod \"maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn\" (UID: \"8d586fea-871e-4f69-bb4b-80d4a8c24a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn" Oct 30 21:52:02.028172 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:02.028134 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d586fea-871e-4f69-bb4b-80d4a8c24a1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn\" (UID: \"8d586fea-871e-4f69-bb4b-80d4a8c24a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn" Oct 30 21:52:02.038262 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:02.038233 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb"] Oct 30 21:52:02.042143 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:02.042120 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1068fa75-rmqfb"] Oct 30 21:52:02.129293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:02.129255 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d586fea-871e-4f69-bb4b-80d4a8c24a1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn\" (UID: \"8d586fea-871e-4f69-bb4b-80d4a8c24a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn" Oct 30 21:52:02.129482 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:02.129420 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h4lc\" (UniqueName: \"kubernetes.io/projected/8d586fea-871e-4f69-bb4b-80d4a8c24a1f-kube-api-access-6h4lc\") pod \"maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn\" (UID: \"8d586fea-871e-4f69-bb4b-80d4a8c24a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn" Oct 30 21:52:02.129542 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:02.129487 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d586fea-871e-4f69-bb4b-80d4a8c24a1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn\" (UID: \"8d586fea-871e-4f69-bb4b-80d4a8c24a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn" Oct 30 21:52:02.130004 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:02.129983 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d586fea-871e-4f69-bb4b-80d4a8c24a1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn\" (UID: \"8d586fea-871e-4f69-bb4b-80d4a8c24a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn" Oct 30 21:52:02.132197 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:02.132170 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d586fea-871e-4f69-bb4b-80d4a8c24a1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn\" (UID: \"8d586fea-871e-4f69-bb4b-80d4a8c24a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn" Oct 30 21:52:02.148022 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:02.147994 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h4lc\" (UniqueName: \"kubernetes.io/projected/8d586fea-871e-4f69-bb4b-80d4a8c24a1f-kube-api-access-6h4lc\") pod \"maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn\" (UID: \"8d586fea-871e-4f69-bb4b-80d4a8c24a1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn" Oct 30 21:52:02.277898 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:02.277804 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn" Oct 30 21:52:02.592174 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:02.592138 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn"] Oct 30 21:52:02.593343 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:52:02.593313 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d586fea_871e_4f69_bb4b_80d4a8c24a1f.slice/crio-e215e54849057d1ce58ba8cd743dfb90592ccb000edfe0290f8274dec4915895 WatchSource:0}: Error finding container e215e54849057d1ce58ba8cd743dfb90592ccb000edfe0290f8274dec4915895: Status 404 returned error can't find the container with id e215e54849057d1ce58ba8cd743dfb90592ccb000edfe0290f8274dec4915895 Oct 30 21:52:03.075242 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:03.075201 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="219457b4-60ac-4501-aeda-f40528bddee1" path="/var/lib/kubelet/pods/219457b4-60ac-4501-aeda-f40528bddee1/volumes" Oct 30 21:52:03.449058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:03.444316 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" podUID="7d9e51cd-e4a7-4944-a1da-23a596732dbb" containerName="main" containerID="cri-o://ab7af3be6c85ed08a618bceeb207a8d5ab5302a47d05f5cb18c9953ecd42881b" gracePeriod=45 Oct 30 21:52:03.503210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:03.503126 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn" event={"ID":"8d586fea-871e-4f69-bb4b-80d4a8c24a1f","Type":"ContainerStarted","Data":"2e1b19aeaf8c889183fd369944cccfcf020c57eee2ff3668dfaa53696102b49c"} Oct 30 21:52:03.503210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:03.503195 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn" event={"ID":"8d586fea-871e-4f69-bb4b-80d4a8c24a1f","Type":"ContainerStarted","Data":"e215e54849057d1ce58ba8cd743dfb90592ccb000edfe0290f8274dec4915895"} Oct 30 21:52:03.575054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:03.569145 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn" podStartSLOduration=2.569124074 podStartE2EDuration="2.569124074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:03.566523371 +0000 UTC m=+3110.687597144" watchObservedRunningTime="2026-10-30 21:52:03.569124074 +0000 UTC m=+3110.690197821" Oct 30 21:52:03.668580 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:03.668285 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" podUID="7d9e51cd-e4a7-4944-a1da-23a596732dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.246:8000/ready\": dial tcp 10.132.0.246:8000: connect: connection refused" Oct 30 21:52:03.668736 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:03.668656 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj"] Oct 30 21:52:03.675089 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:03.675059 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj" Oct 30 21:52:03.693962 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:03.693910 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-be3ada-dockercfg-l6twd\"" Oct 30 21:52:03.717218 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:03.705614 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj"] Oct 30 21:52:03.746698 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:03.745550 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-be3ada\"" Oct 30 21:52:03.784054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:03.776535 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b99d362-edb2-44b1-bfc7-a6688359aff5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj\" (UID: \"7b99d362-edb2-44b1-bfc7-a6688359aff5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj" Oct 30 21:52:03.784054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:03.776658 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbqqx\" (UniqueName: \"kubernetes.io/projected/7b99d362-edb2-44b1-bfc7-a6688359aff5-kube-api-access-jbqqx\") pod \"payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj\" (UID: \"7b99d362-edb2-44b1-bfc7-a6688359aff5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj" Oct 30 21:52:03.882307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:03.882072 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b99d362-edb2-44b1-bfc7-a6688359aff5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj\" (UID: \"7b99d362-edb2-44b1-bfc7-a6688359aff5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj" Oct 30 21:52:03.882307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:03.882210 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbqqx\" (UniqueName: \"kubernetes.io/projected/7b99d362-edb2-44b1-bfc7-a6688359aff5-kube-api-access-jbqqx\") pod \"payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj\" (UID: \"7b99d362-edb2-44b1-bfc7-a6688359aff5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj" Oct 30 21:52:03.891062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:03.883171 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b99d362-edb2-44b1-bfc7-a6688359aff5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj\" (UID: \"7b99d362-edb2-44b1-bfc7-a6688359aff5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj" Oct 30 21:52:03.926199 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:03.925511 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbqqx\" (UniqueName: \"kubernetes.io/projected/7b99d362-edb2-44b1-bfc7-a6688359aff5-kube-api-access-jbqqx\") pod \"payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj\" (UID: \"7b99d362-edb2-44b1-bfc7-a6688359aff5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj" Oct 30 21:52:03.994988 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:03.994954 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj" Oct 30 21:52:04.020717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.020693 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:52:04.087510 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.087470 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-tmp-dir\") pod \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " Oct 30 21:52:04.087973 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.087546 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d9e51cd-e4a7-4944-a1da-23a596732dbb-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9e51cd-e4a7-4944-a1da-23a596732dbb-tls-certs\") pod \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " Oct 30 21:52:04.087973 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.087589 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-home\" (UniqueName: \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-home\") pod \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " Oct 30 21:52:04.087973 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.087624 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-dshm\") pod \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " Oct 30 21:52:04.087973 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.087760 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d9e51cd-e4a7-4944-a1da-23a596732dbb-kube-api-access-wnjz8\" (UniqueName: \"kubernetes.io/projected/7d9e51cd-e4a7-4944-a1da-23a596732dbb-kube-api-access-wnjz8\") pod \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " Oct 30 21:52:04.087973 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.087957 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-model-cache\") pod \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\" (UID: \"7d9e51cd-e4a7-4944-a1da-23a596732dbb\") " Oct 30 21:52:04.089174 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.089145 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-tmp-dir" pod "7d9e51cd-e4a7-4944-a1da-23a596732dbb" (UID: "7d9e51cd-e4a7-4944-a1da-23a596732dbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:04.090396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.090367 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-model-cache" pod "7d9e51cd-e4a7-4944-a1da-23a596732dbb" (UID: "7d9e51cd-e4a7-4944-a1da-23a596732dbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:04.091550 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.091520 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-home" pod "7d9e51cd-e4a7-4944-a1da-23a596732dbb" (UID: "7d9e51cd-e4a7-4944-a1da-23a596732dbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:04.092822 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.092385 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d9e51cd-e4a7-4944-a1da-23a596732dbb-tls-certs" pod "7d9e51cd-e4a7-4944-a1da-23a596732dbb" (UID: "7d9e51cd-e4a7-4944-a1da-23a596732dbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:04.094834 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.094803 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9e51cd-e4a7-4944-a1da-23a596732dbb-kube-api-access-wnjz8" pod "7d9e51cd-e4a7-4944-a1da-23a596732dbb" (UID: "7d9e51cd-e4a7-4944-a1da-23a596732dbb"). InnerVolumeSpecName "kube-api-access-wnjz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:04.095806 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.095780 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-dshm" pod "7d9e51cd-e4a7-4944-a1da-23a596732dbb" (UID: "7d9e51cd-e4a7-4944-a1da-23a596732dbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:04.190045 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.189997 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:52:04.190228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.190054 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:52:04.190228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.190070 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9e51cd-e4a7-4944-a1da-23a596732dbb-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:52:04.190228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.190082 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:52:04.190228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.190095 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9e51cd-e4a7-4944-a1da-23a596732dbb-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:52:04.190228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.190108 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnjz8\" (UniqueName: \"kubernetes.io/projected/7d9e51cd-e4a7-4944-a1da-23a596732dbb-kube-api-access-wnjz8\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:52:04.318624 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.318586 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj"] Oct 30 21:52:04.517168 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.515991 4926 generic.go:356] "Generic (PLEG): container finished" podID="7d9e51cd-e4a7-4944-a1da-23a596732dbb" containerID="ab7af3be6c85ed08a618bceeb207a8d5ab5302a47d05f5cb18c9953ecd42881b" exitCode=0 Oct 30 21:52:04.517168 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.516113 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" event={"ID":"7d9e51cd-e4a7-4944-a1da-23a596732dbb","Type":"ContainerDied","Data":"ab7af3be6c85ed08a618bceeb207a8d5ab5302a47d05f5cb18c9953ecd42881b"} Oct 30 21:52:04.517168 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.516160 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" event={"ID":"7d9e51cd-e4a7-4944-a1da-23a596732dbb","Type":"ContainerDied","Data":"0cdc529ec746db23c2b2a1eb1882dcaeb9e1d8991f72c648007746edeab10553"} Oct 30 21:52:04.517168 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.516185 4926 scope.go:122] "RemoveContainer" containerID="ab7af3be6c85ed08a618bceeb207a8d5ab5302a47d05f5cb18c9953ecd42881b" Oct 30 21:52:04.517168 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.516381 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl" Oct 30 21:52:04.521523 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.521493 4926 generic.go:356] "Generic (PLEG): container finished" podID="8d586fea-871e-4f69-bb4b-80d4a8c24a1f" containerID="2e1b19aeaf8c889183fd369944cccfcf020c57eee2ff3668dfaa53696102b49c" exitCode=0 Oct 30 21:52:04.521736 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.521687 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn" event={"ID":"8d586fea-871e-4f69-bb4b-80d4a8c24a1f","Type":"ContainerDied","Data":"2e1b19aeaf8c889183fd369944cccfcf020c57eee2ff3668dfaa53696102b49c"} Oct 30 21:52:04.523766 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.523738 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj" event={"ID":"7b99d362-edb2-44b1-bfc7-a6688359aff5","Type":"ContainerStarted","Data":"cd0e52ee095d99326a82ea0707ecd299b6722ac37d90df3cd3fc8b8058242d8e"} Oct 30 21:52:04.523930 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.523788 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj" event={"ID":"7b99d362-edb2-44b1-bfc7-a6688359aff5","Type":"ContainerStarted","Data":"eb2203285f96b82237fbb1b3009236aa7e9de4b8e89ad71b3313f9a4bc231e4d"} Oct 30 21:52:04.523930 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.523845 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj" Oct 30 21:52:04.534797 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.534769 4926 scope.go:122] "RemoveContainer" containerID="ab7af3be6c85ed08a618bceeb207a8d5ab5302a47d05f5cb18c9953ecd42881b" Oct 30 21:52:04.535085 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:52:04.535062 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab7af3be6c85ed08a618bceeb207a8d5ab5302a47d05f5cb18c9953ecd42881b\": container with ID starting with ab7af3be6c85ed08a618bceeb207a8d5ab5302a47d05f5cb18c9953ecd42881b not found: ID does not exist" containerID="ab7af3be6c85ed08a618bceeb207a8d5ab5302a47d05f5cb18c9953ecd42881b" Oct 30 21:52:04.535149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.535093 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7af3be6c85ed08a618bceeb207a8d5ab5302a47d05f5cb18c9953ecd42881b"} err="failed to get container status \"ab7af3be6c85ed08a618bceeb207a8d5ab5302a47d05f5cb18c9953ecd42881b\": rpc error: code = NotFound desc = could not find container \"ab7af3be6c85ed08a618bceeb207a8d5ab5302a47d05f5cb18c9953ecd42881b\": container with ID starting with ab7af3be6c85ed08a618bceeb207a8d5ab5302a47d05f5cb18c9953ecd42881b not found: ID does not exist" Oct 30 21:52:04.614143 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.614007 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj" podStartSLOduration=1.613988393 podStartE2EDuration="1.613988393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:04.611941783 +0000 UTC m=+3111.733015529" watchObservedRunningTime="2026-10-30 21:52:04.613988393 +0000 UTC m=+3111.735062140" Oct 30 21:52:04.649929 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.649892 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl"] Oct 30 21:52:04.659381 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:04.659349 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-908469/sub-test-model-908469-kserve-5b74b5c669-zwhbl"] Oct 30 21:52:05.017309 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:05.017277 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d9e51cd-e4a7-4944-a1da-23a596732dbb" path="/var/lib/kubelet/pods/7d9e51cd-e4a7-4944-a1da-23a596732dbb/volumes" Oct 30 21:52:05.718147 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:05.718079 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn" Oct 30 21:52:05.812441 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:05.812406 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d586fea-871e-4f69-bb4b-80d4a8c24a1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d586fea-871e-4f69-bb4b-80d4a8c24a1f-maas-api-xks-ca\") pod \"8d586fea-871e-4f69-bb4b-80d4a8c24a1f\" (UID: \"8d586fea-871e-4f69-bb4b-80d4a8c24a1f\") " Oct 30 21:52:05.812647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:05.812467 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d586fea-871e-4f69-bb4b-80d4a8c24a1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d586fea-871e-4f69-bb4b-80d4a8c24a1f-maas-api-openshift-ca\") pod \"8d586fea-871e-4f69-bb4b-80d4a8c24a1f\" (UID: \"8d586fea-871e-4f69-bb4b-80d4a8c24a1f\") " Oct 30 21:52:05.812647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:05.812498 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d586fea-871e-4f69-bb4b-80d4a8c24a1f-kube-api-access-6h4lc\" (UniqueName: \"kubernetes.io/projected/8d586fea-871e-4f69-bb4b-80d4a8c24a1f-kube-api-access-6h4lc\") pod \"8d586fea-871e-4f69-bb4b-80d4a8c24a1f\" (UID: \"8d586fea-871e-4f69-bb4b-80d4a8c24a1f\") " Oct 30 21:52:05.812869 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:05.812842 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d586fea-871e-4f69-bb4b-80d4a8c24a1f-maas-api-openshift-ca" pod "8d586fea-871e-4f69-bb4b-80d4a8c24a1f" (UID: "8d586fea-871e-4f69-bb4b-80d4a8c24a1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:05.813111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:05.813090 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d586fea-871e-4f69-bb4b-80d4a8c24a1f-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:52:05.815317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:05.815291 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d586fea-871e-4f69-bb4b-80d4a8c24a1f-maas-api-xks-ca" pod "8d586fea-871e-4f69-bb4b-80d4a8c24a1f" (UID: "8d586fea-871e-4f69-bb4b-80d4a8c24a1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:05.815431 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:05.815359 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d586fea-871e-4f69-bb4b-80d4a8c24a1f-kube-api-access-6h4lc" pod "8d586fea-871e-4f69-bb4b-80d4a8c24a1f" (UID: "8d586fea-871e-4f69-bb4b-80d4a8c24a1f"). InnerVolumeSpecName "kube-api-access-6h4lc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:05.914435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:05.914399 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d586fea-871e-4f69-bb4b-80d4a8c24a1f-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:52:05.914435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:05.914433 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h4lc\" (UniqueName: \"kubernetes.io/projected/8d586fea-871e-4f69-bb4b-80d4a8c24a1f-kube-api-access-6h4lc\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:52:06.542627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:06.542595 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn" Oct 30 21:52:06.542821 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:06.542626 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn" event={"ID":"8d586fea-871e-4f69-bb4b-80d4a8c24a1f","Type":"ContainerDied","Data":"e215e54849057d1ce58ba8cd743dfb90592ccb000edfe0290f8274dec4915895"} Oct 30 21:52:06.542821 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:06.542681 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e215e54849057d1ce58ba8cd743dfb90592ccb000edfe0290f8274dec4915895" Oct 30 21:52:07.489156 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:52:07.489117 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:52:12.374849 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:12.374620 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" podUID="875f580c-1467-4e76-9818-0d0154042c3f" containerName="main" containerID="cri-o://2830d94cda182db920ce0ad8a29304cd9f7bc5c564c09db17fec346136587c09" gracePeriod=45 Oct 30 21:52:12.654116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:12.653302 4926 generic.go:356] "Generic (PLEG): container finished" podID="875f580c-1467-4e76-9818-0d0154042c3f" containerID="2830d94cda182db920ce0ad8a29304cd9f7bc5c564c09db17fec346136587c09" exitCode=0 Oct 30 21:52:12.654116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:12.653613 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" event={"ID":"875f580c-1467-4e76-9818-0d0154042c3f","Type":"ContainerDied","Data":"2830d94cda182db920ce0ad8a29304cd9f7bc5c564c09db17fec346136587c09"} Oct 30 21:52:12.828144 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:12.828116 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:52:12.916773 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:12.916741 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/875f580c-1467-4e76-9818-0d0154042c3f-tls-certs\" (UniqueName: \"kubernetes.io/secret/875f580c-1467-4e76-9818-0d0154042c3f-tls-certs\") pod \"875f580c-1467-4e76-9818-0d0154042c3f\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " Oct 30 21:52:12.916971 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:12.916813 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-home\" (UniqueName: \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-home\") pod \"875f580c-1467-4e76-9818-0d0154042c3f\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " Oct 30 21:52:12.916971 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:12.916850 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-dshm\") pod \"875f580c-1467-4e76-9818-0d0154042c3f\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " Oct 30 21:52:12.916971 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:12.916892 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/875f580c-1467-4e76-9818-0d0154042c3f-kube-api-access-m8q59\" (UniqueName: \"kubernetes.io/projected/875f580c-1467-4e76-9818-0d0154042c3f-kube-api-access-m8q59\") pod \"875f580c-1467-4e76-9818-0d0154042c3f\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " Oct 30 21:52:12.916971 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:12.916927 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-model-cache\") pod \"875f580c-1467-4e76-9818-0d0154042c3f\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " Oct 30 21:52:12.917253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:12.917147 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-home" pod "875f580c-1467-4e76-9818-0d0154042c3f" (UID: "875f580c-1467-4e76-9818-0d0154042c3f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:12.917253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:12.917235 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-tmp-dir\") pod \"875f580c-1467-4e76-9818-0d0154042c3f\" (UID: \"875f580c-1467-4e76-9818-0d0154042c3f\") " Oct 30 21:52:12.917745 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:12.917714 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-tmp-dir" pod "875f580c-1467-4e76-9818-0d0154042c3f" (UID: "875f580c-1467-4e76-9818-0d0154042c3f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:12.917891 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:12.917878 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:52:12.917959 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:12.917896 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:52:12.918082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:12.918059 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-model-cache" pod "875f580c-1467-4e76-9818-0d0154042c3f" (UID: "875f580c-1467-4e76-9818-0d0154042c3f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:12.919613 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:12.919584 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/875f580c-1467-4e76-9818-0d0154042c3f-kube-api-access-m8q59" pod "875f580c-1467-4e76-9818-0d0154042c3f" (UID: "875f580c-1467-4e76-9818-0d0154042c3f"). InnerVolumeSpecName "kube-api-access-m8q59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:12.919740 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:12.919722 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-dshm" pod "875f580c-1467-4e76-9818-0d0154042c3f" (UID: "875f580c-1467-4e76-9818-0d0154042c3f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:52:12.919798 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:12.919764 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/875f580c-1467-4e76-9818-0d0154042c3f-tls-certs" pod "875f580c-1467-4e76-9818-0d0154042c3f" (UID: "875f580c-1467-4e76-9818-0d0154042c3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:13.018958 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:13.018927 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/875f580c-1467-4e76-9818-0d0154042c3f-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:52:13.018958 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:13.018955 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:52:13.019205 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:13.018966 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8q59\" (UniqueName: \"kubernetes.io/projected/875f580c-1467-4e76-9818-0d0154042c3f-kube-api-access-m8q59\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:52:13.019205 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:13.018977 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/875f580c-1467-4e76-9818-0d0154042c3f-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:52:13.662853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:13.662822 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" Oct 30 21:52:13.665523 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:13.662863 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv" event={"ID":"875f580c-1467-4e76-9818-0d0154042c3f","Type":"ContainerDied","Data":"e1a5605ffbd3c47facd1ab4abb31174cbcda909c12e658e792474c193784250e"} Oct 30 21:52:13.665523 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:13.662915 4926 scope.go:122] "RemoveContainer" containerID="2830d94cda182db920ce0ad8a29304cd9f7bc5c564c09db17fec346136587c09" Oct 30 21:52:13.713324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:13.713211 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv"] Oct 30 21:52:13.724995 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:13.724959 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-95692f/sub-test-model-95692f-kserve-6dbcb47d55-prlxv"] Oct 30 21:52:15.018798 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:15.018763 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="875f580c-1467-4e76-9818-0d0154042c3f" path="/var/lib/kubelet/pods/875f580c-1467-4e76-9818-0d0154042c3f/volumes" Oct 30 21:52:15.537568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:15.537535 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj" Oct 30 21:52:21.058463 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.058343 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t"] Oct 30 21:52:21.059638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.059607 4926 cpu_manager.go:414] "containerMap: removing container" podUID="7d9e51cd-e4a7-4944-a1da-23a596732dbb" containerName="main" Oct 30 21:52:21.059638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.059636 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d9e51cd-e4a7-4944-a1da-23a596732dbb" containerName="main" Oct 30 21:52:21.059830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.059697 4926 cpu_manager.go:414] "containerMap: removing container" podUID="875f580c-1467-4e76-9818-0d0154042c3f" containerName="main" Oct 30 21:52:21.059830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.059706 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="875f580c-1467-4e76-9818-0d0154042c3f" containerName="main" Oct 30 21:52:21.059830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.059716 4926 cpu_manager.go:414] "containerMap: removing container" podUID="8d586fea-871e-4f69-bb4b-80d4a8c24a1f" containerName="revoke-keys" Oct 30 21:52:21.059830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.059724 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d586fea-871e-4f69-bb4b-80d4a8c24a1f" containerName="revoke-keys" Oct 30 21:52:21.060057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.059925 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="875f580c-1467-4e76-9818-0d0154042c3f" containerName="main" Oct 30 21:52:21.060057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.059942 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d586fea-871e-4f69-bb4b-80d4a8c24a1f" containerName="revoke-keys" Oct 30 21:52:21.060057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.059961 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d9e51cd-e4a7-4944-a1da-23a596732dbb" containerName="main" Oct 30 21:52:21.065384 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.065361 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.091151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.091110 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f8137c-openshift-default-dockercfg-cx66c\"" Oct 30 21:52:21.102616 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.102559 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-podinfo\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.102972 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.102943 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-workload-socket\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.103198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.103167 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-workload-certs\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.103353 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.103334 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-token\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.103479 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.103460 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtw7d\" (UniqueName: \"kubernetes.io/projected/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-kube-api-access-rtw7d\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.103797 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.103775 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-data\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.104007 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.103987 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-credential-socket\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.105116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.104866 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-envoy\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.105116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.104957 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istiod-ca-cert\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.106188 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.105723 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t"] Oct 30 21:52:21.232056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.231646 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-data\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.232056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.231749 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-credential-socket\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.232056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.231780 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-envoy\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.232056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.231811 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istiod-ca-cert\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.232056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.231957 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-podinfo\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.232056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.232017 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-workload-socket\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.232568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.232114 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-workload-certs\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.232568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.232147 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-token\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.232568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.232174 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtw7d\" (UniqueName: \"kubernetes.io/projected/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-kube-api-access-rtw7d\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.234104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.234074 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-data\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.234590 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.234565 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-workload-socket\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.234968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.234948 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-workload-certs\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.273470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.273429 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-credential-socket\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.274407 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.274383 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istiod-ca-cert\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.298055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.277334 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-podinfo\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.298055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.277608 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-envoy\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.308122 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.308023 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-token\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.327871 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.327794 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtw7d\" (UniqueName: \"kubernetes.io/projected/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-kube-api-access-rtw7d\") pod \"e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.406494 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.406229 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:21.748993 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.748959 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t"] Oct 30 21:52:21.751510 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:52:21.751481 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ee4f20_47b0_4c7a_9ad1_ec970173c3ee.slice/crio-d431cbbbcf01501a3567817c414081b6f34db1f43ce7cc9db79576a6ecaa712a WatchSource:0}: Error finding container d431cbbbcf01501a3567817c414081b6f34db1f43ce7cc9db79576a6ecaa712a: Status 404 returned error can't find the container with id d431cbbbcf01501a3567817c414081b6f34db1f43ce7cc9db79576a6ecaa712a Oct 30 21:52:21.755608 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.755554 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:52:21.755720 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.755659 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:52:21.755720 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.755697 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:52:21.828244 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:21.828201 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" event={"ID":"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee","Type":"ContainerStarted","Data":"d431cbbbcf01501a3567817c414081b6f34db1f43ce7cc9db79576a6ecaa712a"} Oct 30 21:52:22.841364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:22.841320 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" event={"ID":"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee","Type":"ContainerStarted","Data":"db7bc3b531fa7b313086aaf969b710e52a1e7b3b6ece5c8f378e2767ccef200e"} Oct 30 21:52:22.913482 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:22.913402 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" podStartSLOduration=1.913381896 podStartE2EDuration="1.913381896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:22.899987931 +0000 UTC m=+3130.021061676" watchObservedRunningTime="2026-10-30 21:52:22.913381896 +0000 UTC m=+3130.034455654" Oct 30 21:52:23.407936 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:23.407897 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:23.414810 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:23.414774 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:23.850195 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:23.850156 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:23.856135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:23.855506 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:52:23.963348 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:23.963204 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": read tcp 10.132.0.2:52266->10.132.1.14:15021: read: connection reset by peer" start-of-body= Oct 30 21:52:23.963348 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:23.963277 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" podUID="e89edd0d-00f1-4a05-baa6-b053af519b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": read tcp 10.132.0.2:52266->10.132.1.14:15021: read: connection reset by peer" Oct 30 21:52:24.859906 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:24.859867 4926 generic.go:356] "Generic (PLEG): container finished" podID="e89edd0d-00f1-4a05-baa6-b053af519b55" containerID="97dbc86ea10608782ca964f1a69998158ded69fbc3ba0cd373748be6457c0284" exitCode=0 Oct 30 21:52:24.860381 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:24.859938 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" event={"ID":"e89edd0d-00f1-4a05-baa6-b053af519b55","Type":"ContainerDied","Data":"97dbc86ea10608782ca964f1a69998158ded69fbc3ba0cd373748be6457c0284"} Oct 30 21:52:24.860640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:24.860625 4926 scope.go:122] "RemoveContainer" containerID="97dbc86ea10608782ca964f1a69998158ded69fbc3ba0cd373748be6457c0284" Oct 30 21:52:24.862286 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:24.862257 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:52:24.862362 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:24.862327 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:52:24.862407 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:24.862359 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:52:25.894738 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:25.894701 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" event={"ID":"e89edd0d-00f1-4a05-baa6-b053af519b55","Type":"ContainerStarted","Data":"4e744ecfcc14b3caca4d66b13cf5fca50d17a959f389770b10003d575266fb6b"} Oct 30 21:52:26.637061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:26.636496 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:52:26.637061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:26.636632 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 21:52:26.637061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:26.636696 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" podUID="e89edd0d-00f1-4a05-baa6-b053af519b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 21:52:27.216103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:27.216059 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq"] Oct 30 21:52:27.220368 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:27.220343 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq" Oct 30 21:52:27.240927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:27.240893 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq"] Oct 30 21:52:27.245435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:27.245406 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f8137c\"" Oct 30 21:52:27.245651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:27.245447 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f8137c-dockercfg-mw9qf\"" Oct 30 21:52:27.308826 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:27.308788 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba871578-1186-44c3-842d-5cfc5baa7f35-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq\" (UID: \"ba871578-1186-44c3-842d-5cfc5baa7f35\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq" Oct 30 21:52:27.309085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:27.308918 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc55g\" (UniqueName: \"kubernetes.io/projected/ba871578-1186-44c3-842d-5cfc5baa7f35-kube-api-access-qc55g\") pod \"payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq\" (UID: \"ba871578-1186-44c3-842d-5cfc5baa7f35\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq" Oct 30 21:52:27.416055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:27.413884 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc55g\" (UniqueName: \"kubernetes.io/projected/ba871578-1186-44c3-842d-5cfc5baa7f35-kube-api-access-qc55g\") pod \"payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq\" (UID: \"ba871578-1186-44c3-842d-5cfc5baa7f35\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq" Oct 30 21:52:27.416055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:27.414208 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba871578-1186-44c3-842d-5cfc5baa7f35-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq\" (UID: \"ba871578-1186-44c3-842d-5cfc5baa7f35\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq" Oct 30 21:52:27.416055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:27.415290 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba871578-1186-44c3-842d-5cfc5baa7f35-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq\" (UID: \"ba871578-1186-44c3-842d-5cfc5baa7f35\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq" Oct 30 21:52:27.468105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:27.467686 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc55g\" (UniqueName: \"kubernetes.io/projected/ba871578-1186-44c3-842d-5cfc5baa7f35-kube-api-access-qc55g\") pod \"payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq\" (UID: \"ba871578-1186-44c3-842d-5cfc5baa7f35\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq" Oct 30 21:52:27.538356 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:27.538297 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq" Oct 30 21:52:27.635832 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:27.635749 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 21:52:27.635832 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:27.635804 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" podUID="e89edd0d-00f1-4a05-baa6-b053af519b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 21:52:27.961125 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:27.961091 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq"] Oct 30 21:52:27.962429 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:52:27.962256 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba871578_1186_44c3_842d_5cfc5baa7f35.slice/crio-c94629ee32d31150b44e7ef5f36e2d5f47ec4702bda7c3d7bbb1cc724acc840e WatchSource:0}: Error finding container c94629ee32d31150b44e7ef5f36e2d5f47ec4702bda7c3d7bbb1cc724acc840e: Status 404 returned error can't find the container with id c94629ee32d31150b44e7ef5f36e2d5f47ec4702bda7c3d7bbb1cc724acc840e Oct 30 21:52:28.635914 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:28.635875 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 21:52:28.636396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:28.635932 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" podUID="e89edd0d-00f1-4a05-baa6-b053af519b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 21:52:28.930108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:28.930064 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq" event={"ID":"ba871578-1186-44c3-842d-5cfc5baa7f35","Type":"ContainerStarted","Data":"d9521d8beacf8f020119c7081cf59ccaf04f04689b512e13ae13619162531999"} Oct 30 21:52:28.930322 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:28.930296 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq" Oct 30 21:52:28.930396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:28.930342 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq" event={"ID":"ba871578-1186-44c3-842d-5cfc5baa7f35","Type":"ContainerStarted","Data":"c94629ee32d31150b44e7ef5f36e2d5f47ec4702bda7c3d7bbb1cc724acc840e"} Oct 30 21:52:29.000053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:28.999843 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq" podStartSLOduration=1.999822445 podStartE2EDuration="1.999822445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:28.98929683 +0000 UTC m=+3136.110370579" watchObservedRunningTime="2026-10-30 21:52:28.999822445 +0000 UTC m=+3136.120896193" Oct 30 21:52:29.635896 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:29.635858 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 21:52:29.636126 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:29.635917 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" podUID="e89edd0d-00f1-4a05-baa6-b053af519b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 21:52:30.636450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:30.636409 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 21:52:30.636850 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:30.636473 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" podUID="e89edd0d-00f1-4a05-baa6-b053af519b55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 21:52:31.641067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:31.641019 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:52:31.641503 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:31.641290 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:52:31.642105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:31.642087 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:52:37.488110 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:52:37.488011 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:52:39.725060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:39.724984 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:52:39.942045 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:39.942007 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq" Oct 30 21:52:49.816928 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:49.816883 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n"] Oct 30 21:52:49.821319 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:49.821293 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n" Oct 30 21:52:49.837951 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:49.837917 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n"] Oct 30 21:52:49.843507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:49.843472 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:52:49.843507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:49.843493 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:52:49.884788 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:49.884754 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhs5s\" (UniqueName: \"kubernetes.io/projected/99ee8ed4-52da-4fcb-aa6d-bb16e6765985-kube-api-access-jhs5s\") pod \"maas-api-revoke-keys-e2e-ipp-a-203209-6n95n\" (UID: \"99ee8ed4-52da-4fcb-aa6d-bb16e6765985\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n" Oct 30 21:52:49.884985 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:49.884880 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99ee8ed4-52da-4fcb-aa6d-bb16e6765985-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-203209-6n95n\" (UID: \"99ee8ed4-52da-4fcb-aa6d-bb16e6765985\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n" Oct 30 21:52:49.884985 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:49.884976 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99ee8ed4-52da-4fcb-aa6d-bb16e6765985-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-203209-6n95n\" (UID: \"99ee8ed4-52da-4fcb-aa6d-bb16e6765985\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n" Oct 30 21:52:49.986225 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:49.986190 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhs5s\" (UniqueName: \"kubernetes.io/projected/99ee8ed4-52da-4fcb-aa6d-bb16e6765985-kube-api-access-jhs5s\") pod \"maas-api-revoke-keys-e2e-ipp-a-203209-6n95n\" (UID: \"99ee8ed4-52da-4fcb-aa6d-bb16e6765985\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n" Oct 30 21:52:49.986443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:49.986279 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99ee8ed4-52da-4fcb-aa6d-bb16e6765985-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-203209-6n95n\" (UID: \"99ee8ed4-52da-4fcb-aa6d-bb16e6765985\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n" Oct 30 21:52:49.986443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:49.986345 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99ee8ed4-52da-4fcb-aa6d-bb16e6765985-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-203209-6n95n\" (UID: \"99ee8ed4-52da-4fcb-aa6d-bb16e6765985\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n" Oct 30 21:52:49.986979 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:49.986952 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99ee8ed4-52da-4fcb-aa6d-bb16e6765985-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-203209-6n95n\" (UID: \"99ee8ed4-52da-4fcb-aa6d-bb16e6765985\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n" Oct 30 21:52:49.988833 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:49.988806 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99ee8ed4-52da-4fcb-aa6d-bb16e6765985-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-203209-6n95n\" (UID: \"99ee8ed4-52da-4fcb-aa6d-bb16e6765985\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n" Oct 30 21:52:50.005401 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:50.005378 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhs5s\" (UniqueName: \"kubernetes.io/projected/99ee8ed4-52da-4fcb-aa6d-bb16e6765985-kube-api-access-jhs5s\") pod \"maas-api-revoke-keys-e2e-ipp-a-203209-6n95n\" (UID: \"99ee8ed4-52da-4fcb-aa6d-bb16e6765985\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n" Oct 30 21:52:50.028267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:50.028230 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0de63b5c-8lrmf"] Oct 30 21:52:50.032160 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:50.032138 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0de63b5c-8lrmf"] Oct 30 21:52:50.135612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:50.135530 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n" Oct 30 21:52:50.448631 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:50.448600 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n"] Oct 30 21:52:50.449465 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:52:50.449439 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ee8ed4_52da_4fcb_aa6d_bb16e6765985.slice/crio-1f50a32428f9efc02e6531b7ffa8e88185de2f99dfa812d57410705971212802 WatchSource:0}: Error finding container 1f50a32428f9efc02e6531b7ffa8e88185de2f99dfa812d57410705971212802: Status 404 returned error can't find the container with id 1f50a32428f9efc02e6531b7ffa8e88185de2f99dfa812d57410705971212802 Oct 30 21:52:51.017008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:51.016965 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="034be202-cb88-4f23-8433-bc1f7bea9573" path="/var/lib/kubelet/pods/034be202-cb88-4f23-8433-bc1f7bea9573/volumes" Oct 30 21:52:51.153335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:51.153233 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n" event={"ID":"99ee8ed4-52da-4fcb-aa6d-bb16e6765985","Type":"ContainerStarted","Data":"2b3a5cb9925ee58a598a0acc886a72e483e1b3d1a89a5d73f0b277a213c87c90"} Oct 30 21:52:51.153335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:51.153304 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n" event={"ID":"99ee8ed4-52da-4fcb-aa6d-bb16e6765985","Type":"ContainerStarted","Data":"1f50a32428f9efc02e6531b7ffa8e88185de2f99dfa812d57410705971212802"} Oct 30 21:52:51.192839 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:51.192773 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n" podStartSLOduration=2.192753758 podStartE2EDuration="2.192753758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:51.18924095 +0000 UTC m=+3158.310314697" watchObservedRunningTime="2026-10-30 21:52:51.192753758 +0000 UTC m=+3158.313827505" Oct 30 21:52:52.176652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:52.176611 4926 generic.go:356] "Generic (PLEG): container finished" podID="99ee8ed4-52da-4fcb-aa6d-bb16e6765985" containerID="2b3a5cb9925ee58a598a0acc886a72e483e1b3d1a89a5d73f0b277a213c87c90" exitCode=0 Oct 30 21:52:52.177283 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:52.176873 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n" event={"ID":"99ee8ed4-52da-4fcb-aa6d-bb16e6765985","Type":"ContainerDied","Data":"2b3a5cb9925ee58a598a0acc886a72e483e1b3d1a89a5d73f0b277a213c87c90"} Oct 30 21:52:53.048260 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:53.048220 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn"] Oct 30 21:52:53.052585 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:53.052557 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1d376-62mvn"] Oct 30 21:52:53.358346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:53.358316 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n" Oct 30 21:52:53.437681 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:53.437645 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99ee8ed4-52da-4fcb-aa6d-bb16e6765985-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99ee8ed4-52da-4fcb-aa6d-bb16e6765985-maas-api-xks-ca\") pod \"99ee8ed4-52da-4fcb-aa6d-bb16e6765985\" (UID: \"99ee8ed4-52da-4fcb-aa6d-bb16e6765985\") " Oct 30 21:52:53.437870 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:53.437857 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99ee8ed4-52da-4fcb-aa6d-bb16e6765985-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99ee8ed4-52da-4fcb-aa6d-bb16e6765985-maas-api-openshift-ca\") pod \"99ee8ed4-52da-4fcb-aa6d-bb16e6765985\" (UID: \"99ee8ed4-52da-4fcb-aa6d-bb16e6765985\") " Oct 30 21:52:53.438001 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:53.437985 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ee8ed4-52da-4fcb-aa6d-bb16e6765985-kube-api-access-jhs5s\" (UniqueName: \"kubernetes.io/projected/99ee8ed4-52da-4fcb-aa6d-bb16e6765985-kube-api-access-jhs5s\") pod \"99ee8ed4-52da-4fcb-aa6d-bb16e6765985\" (UID: \"99ee8ed4-52da-4fcb-aa6d-bb16e6765985\") " Oct 30 21:52:53.438435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:53.438288 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ee8ed4-52da-4fcb-aa6d-bb16e6765985-maas-api-openshift-ca" pod "99ee8ed4-52da-4fcb-aa6d-bb16e6765985" (UID: "99ee8ed4-52da-4fcb-aa6d-bb16e6765985"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:52:53.438658 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:53.438617 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99ee8ed4-52da-4fcb-aa6d-bb16e6765985-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:52:53.440164 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:53.440138 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99ee8ed4-52da-4fcb-aa6d-bb16e6765985-maas-api-xks-ca" pod "99ee8ed4-52da-4fcb-aa6d-bb16e6765985" (UID: "99ee8ed4-52da-4fcb-aa6d-bb16e6765985"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:52:53.440361 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:53.440337 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ee8ed4-52da-4fcb-aa6d-bb16e6765985-kube-api-access-jhs5s" pod "99ee8ed4-52da-4fcb-aa6d-bb16e6765985" (UID: "99ee8ed4-52da-4fcb-aa6d-bb16e6765985"). InnerVolumeSpecName "kube-api-access-jhs5s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:52:53.539943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:53.539902 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhs5s\" (UniqueName: \"kubernetes.io/projected/99ee8ed4-52da-4fcb-aa6d-bb16e6765985-kube-api-access-jhs5s\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:52:53.539943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:53.539939 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99ee8ed4-52da-4fcb-aa6d-bb16e6765985-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:52:54.198874 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:54.198840 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n" Oct 30 21:52:54.199113 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:54.198875 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n" event={"ID":"99ee8ed4-52da-4fcb-aa6d-bb16e6765985","Type":"ContainerDied","Data":"1f50a32428f9efc02e6531b7ffa8e88185de2f99dfa812d57410705971212802"} Oct 30 21:52:54.199113 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:54.198925 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f50a32428f9efc02e6531b7ffa8e88185de2f99dfa812d57410705971212802" Oct 30 21:52:55.017097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:52:55.017060 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d586fea-871e-4f69-bb4b-80d4a8c24a1f" path="/var/lib/kubelet/pods/8d586fea-871e-4f69-bb4b-80d4a8c24a1f/volumes" Oct 30 21:53:01.035701 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:01.035659 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv"] Oct 30 21:53:01.045976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:01.045941 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bcded31-fd2wv"] Oct 30 21:53:03.019472 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:03.019409 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6efe2df4-eb09-4380-be6f-96538c9bea6a" path="/var/lib/kubelet/pods/6efe2df4-eb09-4380-be6f-96538c9bea6a/volumes" Oct 30 21:53:05.297296 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:05.297259 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf"] Oct 30 21:53:05.297913 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:05.297897 4926 cpu_manager.go:414] "containerMap: removing container" podUID="99ee8ed4-52da-4fcb-aa6d-bb16e6765985" containerName="revoke-keys" Oct 30 21:53:05.297913 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:05.297914 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ee8ed4-52da-4fcb-aa6d-bb16e6765985" containerName="revoke-keys" Oct 30 21:53:05.298079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:05.298067 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="99ee8ed4-52da-4fcb-aa6d-bb16e6765985" containerName="revoke-keys" Oct 30 21:53:05.303292 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:05.303275 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf" Oct 30 21:53:05.424154 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:05.424112 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b852d25-3186-476d-90be-f1732a4d85e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf\" (UID: \"4b852d25-3186-476d-90be-f1732a4d85e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf" Oct 30 21:53:05.424354 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:05.424224 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfr2n\" (UniqueName: \"kubernetes.io/projected/4b852d25-3186-476d-90be-f1732a4d85e2-kube-api-access-tfr2n\") pod \"maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf\" (UID: \"4b852d25-3186-476d-90be-f1732a4d85e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf" Oct 30 21:53:05.424462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:05.424446 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b852d25-3186-476d-90be-f1732a4d85e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf\" (UID: \"4b852d25-3186-476d-90be-f1732a4d85e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf" Oct 30 21:53:05.461416 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:05.461380 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf"] Oct 30 21:53:05.467153 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:05.467115 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:53:05.467153 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:05.467140 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:53:05.526337 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:05.526303 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b852d25-3186-476d-90be-f1732a4d85e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf\" (UID: \"4b852d25-3186-476d-90be-f1732a4d85e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf" Oct 30 21:53:05.526514 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:05.526397 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b852d25-3186-476d-90be-f1732a4d85e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf\" (UID: \"4b852d25-3186-476d-90be-f1732a4d85e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf" Oct 30 21:53:05.526514 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:05.526449 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfr2n\" (UniqueName: \"kubernetes.io/projected/4b852d25-3186-476d-90be-f1732a4d85e2-kube-api-access-tfr2n\") pod \"maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf\" (UID: \"4b852d25-3186-476d-90be-f1732a4d85e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf" Oct 30 21:53:05.527200 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:05.527170 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b852d25-3186-476d-90be-f1732a4d85e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf\" (UID: \"4b852d25-3186-476d-90be-f1732a4d85e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf" Oct 30 21:53:05.528982 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:05.528959 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b852d25-3186-476d-90be-f1732a4d85e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf\" (UID: \"4b852d25-3186-476d-90be-f1732a4d85e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf" Oct 30 21:53:05.548267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:05.548196 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfr2n\" (UniqueName: \"kubernetes.io/projected/4b852d25-3186-476d-90be-f1732a4d85e2-kube-api-access-tfr2n\") pod \"maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf\" (UID: \"4b852d25-3186-476d-90be-f1732a4d85e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf" Oct 30 21:53:05.618309 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:05.618269 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf" Oct 30 21:53:05.931920 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:05.931886 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf"] Oct 30 21:53:05.933211 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:53:05.933182 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b852d25_3186_476d_90be_f1732a4d85e2.slice/crio-7098a74f1a501b5d349242674d81327c43fbd5f5296fa042b19ae576c15edac0 WatchSource:0}: Error finding container 7098a74f1a501b5d349242674d81327c43fbd5f5296fa042b19ae576c15edac0: Status 404 returned error can't find the container with id 7098a74f1a501b5d349242674d81327c43fbd5f5296fa042b19ae576c15edac0 Oct 30 21:53:06.325580 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:06.324542 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf" event={"ID":"4b852d25-3186-476d-90be-f1732a4d85e2","Type":"ContainerStarted","Data":"08f18fb9ce15168cfe99628bdf7eb17c9ad239e88093f35380074816e3144fcd"} Oct 30 21:53:06.325580 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:06.324898 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf" event={"ID":"4b852d25-3186-476d-90be-f1732a4d85e2","Type":"ContainerStarted","Data":"7098a74f1a501b5d349242674d81327c43fbd5f5296fa042b19ae576c15edac0"} Oct 30 21:53:06.368651 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:06.368596 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf" podStartSLOduration=1.368580953 podStartE2EDuration="1.368580953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:06.364667943 +0000 UTC m=+3173.485741675" watchObservedRunningTime="2026-10-30 21:53:06.368580953 +0000 UTC m=+3173.489654698" Oct 30 21:53:07.335249 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:07.335163 4926 generic.go:356] "Generic (PLEG): container finished" podID="4b852d25-3186-476d-90be-f1732a4d85e2" containerID="08f18fb9ce15168cfe99628bdf7eb17c9ad239e88093f35380074816e3144fcd" exitCode=0 Oct 30 21:53:07.335656 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:07.335237 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf" event={"ID":"4b852d25-3186-476d-90be-f1732a4d85e2","Type":"ContainerDied","Data":"08f18fb9ce15168cfe99628bdf7eb17c9ad239e88093f35380074816e3144fcd"} Oct 30 21:53:07.486230 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:53:07.486184 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:53:08.506837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:08.506809 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf" Oct 30 21:53:08.571905 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:08.571864 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b852d25-3186-476d-90be-f1732a4d85e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b852d25-3186-476d-90be-f1732a4d85e2-maas-api-xks-ca\") pod \"4b852d25-3186-476d-90be-f1732a4d85e2\" (UID: \"4b852d25-3186-476d-90be-f1732a4d85e2\") " Oct 30 21:53:08.572106 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:08.572002 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b852d25-3186-476d-90be-f1732a4d85e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b852d25-3186-476d-90be-f1732a4d85e2-maas-api-openshift-ca\") pod \"4b852d25-3186-476d-90be-f1732a4d85e2\" (UID: \"4b852d25-3186-476d-90be-f1732a4d85e2\") " Oct 30 21:53:08.572220 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:08.572174 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b852d25-3186-476d-90be-f1732a4d85e2-kube-api-access-tfr2n\" (UniqueName: \"kubernetes.io/projected/4b852d25-3186-476d-90be-f1732a4d85e2-kube-api-access-tfr2n\") pod \"4b852d25-3186-476d-90be-f1732a4d85e2\" (UID: \"4b852d25-3186-476d-90be-f1732a4d85e2\") " Oct 30 21:53:08.572472 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:08.572436 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b852d25-3186-476d-90be-f1732a4d85e2-maas-api-openshift-ca" pod "4b852d25-3186-476d-90be-f1732a4d85e2" (UID: "4b852d25-3186-476d-90be-f1732a4d85e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:08.572812 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:08.572791 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b852d25-3186-476d-90be-f1732a4d85e2-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:53:08.574258 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:08.574232 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b852d25-3186-476d-90be-f1732a4d85e2-maas-api-xks-ca" pod "4b852d25-3186-476d-90be-f1732a4d85e2" (UID: "4b852d25-3186-476d-90be-f1732a4d85e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:08.574702 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:08.574684 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b852d25-3186-476d-90be-f1732a4d85e2-kube-api-access-tfr2n" pod "4b852d25-3186-476d-90be-f1732a4d85e2" (UID: "4b852d25-3186-476d-90be-f1732a4d85e2"). InnerVolumeSpecName "kube-api-access-tfr2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:08.674430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:08.674384 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfr2n\" (UniqueName: \"kubernetes.io/projected/4b852d25-3186-476d-90be-f1732a4d85e2-kube-api-access-tfr2n\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:53:08.674430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:08.674425 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b852d25-3186-476d-90be-f1732a4d85e2-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:53:09.353670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:09.353579 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf" event={"ID":"4b852d25-3186-476d-90be-f1732a4d85e2","Type":"ContainerDied","Data":"7098a74f1a501b5d349242674d81327c43fbd5f5296fa042b19ae576c15edac0"} Oct 30 21:53:09.353670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:09.353613 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf" Oct 30 21:53:09.353670 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:09.353637 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7098a74f1a501b5d349242674d81327c43fbd5f5296fa042b19ae576c15edac0" Oct 30 21:53:18.738977 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:18.738938 4926 scope.go:122] "RemoveContainer" containerID="3a6002f00dac67bb409b44fb30a15a09de17474ed59f850de00aedc964178f74" Oct 30 21:53:18.750960 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:18.750936 4926 scope.go:122] "RemoveContainer" containerID="4f44368f926c180adf26a507562d4a8643ecd0fc2fc63b972d9e10c746968321" Oct 30 21:53:18.761608 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:18.761573 4926 scope.go:122] "RemoveContainer" containerID="f037d1f911c9d69ecd76c32e1029cdebc53e30159813bda6c05719dbbf02d958" Oct 30 21:53:18.774163 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:18.774141 4926 scope.go:122] "RemoveContainer" containerID="127ff05bb830c630034cc7270e3e202b184802db0e8f5417efadb66e13ba1694" Oct 30 21:53:18.785025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:18.785003 4926 scope.go:122] "RemoveContainer" containerID="b9e57c58b2fd1a9686f5caa79bc8c5f66421b8be19ae7118a48f64fb8833ddfc" Oct 30 21:53:21.234102 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:21.234061 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4016884f-c6vqr"] Oct 30 21:53:21.234968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:21.234940 4926 cpu_manager.go:414] "containerMap: removing container" podUID="4b852d25-3186-476d-90be-f1732a4d85e2" containerName="revoke-keys" Oct 30 21:53:21.234968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:21.234966 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b852d25-3186-476d-90be-f1732a4d85e2" containerName="revoke-keys" Oct 30 21:53:21.235217 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:21.235202 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b852d25-3186-476d-90be-f1732a4d85e2" containerName="revoke-keys" Oct 30 21:53:21.240847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:21.240820 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4016884f-c6vqr" Oct 30 21:53:21.260210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:21.260179 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4016884f-c6vqr"] Oct 30 21:53:21.263002 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:21.262978 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:53:21.263327 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:21.263302 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:53:21.284068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:21.284001 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40e13d6-fe4d-4ff3-b48d-4e57d1985db8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4016884f-c6vqr\" (UID: \"f40e13d6-fe4d-4ff3-b48d-4e57d1985db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4016884f-c6vqr" Oct 30 21:53:21.284816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:21.284785 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40e13d6-fe4d-4ff3-b48d-4e57d1985db8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4016884f-c6vqr\" (UID: \"f40e13d6-fe4d-4ff3-b48d-4e57d1985db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4016884f-c6vqr" Oct 30 21:53:21.285212 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:21.285190 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qrj5\" (UniqueName: \"kubernetes.io/projected/f40e13d6-fe4d-4ff3-b48d-4e57d1985db8-kube-api-access-8qrj5\") pod \"maas-api-revoke-sub-4016884f-c6vqr\" (UID: \"f40e13d6-fe4d-4ff3-b48d-4e57d1985db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4016884f-c6vqr" Oct 30 21:53:21.388056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:21.387906 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40e13d6-fe4d-4ff3-b48d-4e57d1985db8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4016884f-c6vqr\" (UID: \"f40e13d6-fe4d-4ff3-b48d-4e57d1985db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4016884f-c6vqr" Oct 30 21:53:21.388284 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:21.388098 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qrj5\" (UniqueName: \"kubernetes.io/projected/f40e13d6-fe4d-4ff3-b48d-4e57d1985db8-kube-api-access-8qrj5\") pod \"maas-api-revoke-sub-4016884f-c6vqr\" (UID: \"f40e13d6-fe4d-4ff3-b48d-4e57d1985db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4016884f-c6vqr" Oct 30 21:53:21.388284 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:21.388235 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40e13d6-fe4d-4ff3-b48d-4e57d1985db8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4016884f-c6vqr\" (UID: \"f40e13d6-fe4d-4ff3-b48d-4e57d1985db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4016884f-c6vqr" Oct 30 21:53:21.393065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:21.389113 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40e13d6-fe4d-4ff3-b48d-4e57d1985db8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4016884f-c6vqr\" (UID: \"f40e13d6-fe4d-4ff3-b48d-4e57d1985db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4016884f-c6vqr" Oct 30 21:53:21.397370 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:21.397339 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40e13d6-fe4d-4ff3-b48d-4e57d1985db8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4016884f-c6vqr\" (UID: \"f40e13d6-fe4d-4ff3-b48d-4e57d1985db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4016884f-c6vqr" Oct 30 21:53:21.415369 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:21.415334 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qrj5\" (UniqueName: \"kubernetes.io/projected/f40e13d6-fe4d-4ff3-b48d-4e57d1985db8-kube-api-access-8qrj5\") pod \"maas-api-revoke-sub-4016884f-c6vqr\" (UID: \"f40e13d6-fe4d-4ff3-b48d-4e57d1985db8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4016884f-c6vqr" Oct 30 21:53:21.561889 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:21.561794 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4016884f-c6vqr" Oct 30 21:53:21.954347 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:53:21.954310 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf40e13d6_fe4d_4ff3_b48d_4e57d1985db8.slice/crio-c072f793d1ce993c8c67be2c80523e295bab826500c91607f26910a53ca24c9e WatchSource:0}: Error finding container c072f793d1ce993c8c67be2c80523e295bab826500c91607f26910a53ca24c9e: Status 404 returned error can't find the container with id c072f793d1ce993c8c67be2c80523e295bab826500c91607f26910a53ca24c9e Oct 30 21:53:21.954957 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:21.954930 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4016884f-c6vqr"] Oct 30 21:53:22.499267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:22.499233 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4016884f-c6vqr" event={"ID":"f40e13d6-fe4d-4ff3-b48d-4e57d1985db8","Type":"ContainerStarted","Data":"c072f793d1ce993c8c67be2c80523e295bab826500c91607f26910a53ca24c9e"} Oct 30 21:53:23.508981 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:23.508943 4926 generic.go:356] "Generic (PLEG): container finished" podID="f40e13d6-fe4d-4ff3-b48d-4e57d1985db8" containerID="c10835c1fecbe02f30b2a8844d43afddc7e7a31219118d9529a639369454e759" exitCode=0 Oct 30 21:53:23.509478 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:23.509022 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4016884f-c6vqr" event={"ID":"f40e13d6-fe4d-4ff3-b48d-4e57d1985db8","Type":"ContainerDied","Data":"c10835c1fecbe02f30b2a8844d43afddc7e7a31219118d9529a639369454e759"} Oct 30 21:53:23.758023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:23.757987 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7395-4c98c5"] Oct 30 21:53:23.762711 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:23.762686 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7395-4c98c5" Oct 30 21:53:23.785053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:23.781372 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gxp2m\"" Oct 30 21:53:23.804770 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:23.804740 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7395-4c98c5"] Oct 30 21:53:23.862445 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:23.862410 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp9s9\" (UniqueName: \"kubernetes.io/projected/d9631bf6-2ea3-4adf-84fc-31b954c49a2e-kube-api-access-zp9s9\") pod \"test-curl-7395-4c98c5\" (UID: \"d9631bf6-2ea3-4adf-84fc-31b954c49a2e\") " pod="opendatahub/test-curl-7395-4c98c5" Oct 30 21:53:23.964569 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:23.964521 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp9s9\" (UniqueName: \"kubernetes.io/projected/d9631bf6-2ea3-4adf-84fc-31b954c49a2e-kube-api-access-zp9s9\") pod \"test-curl-7395-4c98c5\" (UID: \"d9631bf6-2ea3-4adf-84fc-31b954c49a2e\") " pod="opendatahub/test-curl-7395-4c98c5" Oct 30 21:53:23.994865 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:23.994824 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp9s9\" (UniqueName: \"kubernetes.io/projected/d9631bf6-2ea3-4adf-84fc-31b954c49a2e-kube-api-access-zp9s9\") pod \"test-curl-7395-4c98c5\" (UID: \"d9631bf6-2ea3-4adf-84fc-31b954c49a2e\") " pod="opendatahub/test-curl-7395-4c98c5" Oct 30 21:53:24.116872 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:24.116772 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7395-4c98c5" Oct 30 21:53:24.451177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:24.451137 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7395-4c98c5"] Oct 30 21:53:24.522573 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:24.522533 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7395-4c98c5" event={"ID":"d9631bf6-2ea3-4adf-84fc-31b954c49a2e","Type":"ContainerStarted","Data":"d7e865d19a69f90df26c6827022ce5b672915c69094644028db8dc8f8449a28b"} Oct 30 21:53:24.711824 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:24.711742 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5"] Oct 30 21:53:24.716532 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:24.716510 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5" Oct 30 21:53:24.732758 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:24.732411 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5"] Oct 30 21:53:24.743898 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:24.742870 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4016884f-c6vqr" Oct 30 21:53:24.906111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:24.905617 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f40e13d6-fe4d-4ff3-b48d-4e57d1985db8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40e13d6-fe4d-4ff3-b48d-4e57d1985db8-maas-api-openshift-ca\") pod \"f40e13d6-fe4d-4ff3-b48d-4e57d1985db8\" (UID: \"f40e13d6-fe4d-4ff3-b48d-4e57d1985db8\") " Oct 30 21:53:24.906111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:24.905681 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f40e13d6-fe4d-4ff3-b48d-4e57d1985db8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40e13d6-fe4d-4ff3-b48d-4e57d1985db8-maas-api-xks-ca\") pod \"f40e13d6-fe4d-4ff3-b48d-4e57d1985db8\" (UID: \"f40e13d6-fe4d-4ff3-b48d-4e57d1985db8\") " Oct 30 21:53:24.906111 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:24.905716 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40e13d6-fe4d-4ff3-b48d-4e57d1985db8-kube-api-access-8qrj5\" (UniqueName: \"kubernetes.io/projected/f40e13d6-fe4d-4ff3-b48d-4e57d1985db8-kube-api-access-8qrj5\") pod \"f40e13d6-fe4d-4ff3-b48d-4e57d1985db8\" (UID: \"f40e13d6-fe4d-4ff3-b48d-4e57d1985db8\") " Oct 30 21:53:24.906434 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:24.906135 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78537147-7ebb-42ef-9295-b8d27a2560fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5\" (UID: \"78537147-7ebb-42ef-9295-b8d27a2560fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5" Oct 30 21:53:24.906434 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:24.906346 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8g6q\" (UniqueName: \"kubernetes.io/projected/78537147-7ebb-42ef-9295-b8d27a2560fc-kube-api-access-n8g6q\") pod \"maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5\" (UID: \"78537147-7ebb-42ef-9295-b8d27a2560fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5" Oct 30 21:53:24.906548 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:24.906468 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78537147-7ebb-42ef-9295-b8d27a2560fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5\" (UID: \"78537147-7ebb-42ef-9295-b8d27a2560fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5" Oct 30 21:53:24.908550 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:24.908517 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40e13d6-fe4d-4ff3-b48d-4e57d1985db8-maas-api-openshift-ca" pod "f40e13d6-fe4d-4ff3-b48d-4e57d1985db8" (UID: "f40e13d6-fe4d-4ff3-b48d-4e57d1985db8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:24.910317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:24.910288 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f40e13d6-fe4d-4ff3-b48d-4e57d1985db8-maas-api-xks-ca" pod "f40e13d6-fe4d-4ff3-b48d-4e57d1985db8" (UID: "f40e13d6-fe4d-4ff3-b48d-4e57d1985db8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:24.910448 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:24.910406 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40e13d6-fe4d-4ff3-b48d-4e57d1985db8-kube-api-access-8qrj5" pod "f40e13d6-fe4d-4ff3-b48d-4e57d1985db8" (UID: "f40e13d6-fe4d-4ff3-b48d-4e57d1985db8"). InnerVolumeSpecName "kube-api-access-8qrj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:25.008298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:25.007407 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8g6q\" (UniqueName: \"kubernetes.io/projected/78537147-7ebb-42ef-9295-b8d27a2560fc-kube-api-access-n8g6q\") pod \"maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5\" (UID: \"78537147-7ebb-42ef-9295-b8d27a2560fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5" Oct 30 21:53:25.008298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:25.007498 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78537147-7ebb-42ef-9295-b8d27a2560fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5\" (UID: \"78537147-7ebb-42ef-9295-b8d27a2560fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5" Oct 30 21:53:25.008298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:25.007569 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78537147-7ebb-42ef-9295-b8d27a2560fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5\" (UID: \"78537147-7ebb-42ef-9295-b8d27a2560fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5" Oct 30 21:53:25.008298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:25.007626 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f40e13d6-fe4d-4ff3-b48d-4e57d1985db8-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:53:25.008298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:25.007636 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f40e13d6-fe4d-4ff3-b48d-4e57d1985db8-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:53:25.008298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:25.007645 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qrj5\" (UniqueName: \"kubernetes.io/projected/f40e13d6-fe4d-4ff3-b48d-4e57d1985db8-kube-api-access-8qrj5\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:53:25.008813 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:25.008307 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78537147-7ebb-42ef-9295-b8d27a2560fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5\" (UID: \"78537147-7ebb-42ef-9295-b8d27a2560fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5" Oct 30 21:53:25.011735 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:25.011702 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78537147-7ebb-42ef-9295-b8d27a2560fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5\" (UID: \"78537147-7ebb-42ef-9295-b8d27a2560fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5" Oct 30 21:53:25.027504 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:25.027471 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8g6q\" (UniqueName: \"kubernetes.io/projected/78537147-7ebb-42ef-9295-b8d27a2560fc-kube-api-access-n8g6q\") pod \"maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5\" (UID: \"78537147-7ebb-42ef-9295-b8d27a2560fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5" Oct 30 21:53:25.041506 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:25.041483 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5" Oct 30 21:53:25.396979 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:25.396933 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5"] Oct 30 21:53:25.533016 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:25.532970 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5" event={"ID":"78537147-7ebb-42ef-9295-b8d27a2560fc","Type":"ContainerStarted","Data":"9c7afeec7fbbc4bc1c5f0a0ac2ee3024877361360a591744859396f30398baf6"} Oct 30 21:53:25.535546 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:25.535514 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7395-4c98c5" event={"ID":"d9631bf6-2ea3-4adf-84fc-31b954c49a2e","Type":"ContainerStarted","Data":"08e57781ed778e81dbf3bcd0ed1fffd21a8618edb905b1a12d1b5e54e5922424"} Oct 30 21:53:25.537103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:25.537083 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4016884f-c6vqr" Oct 30 21:53:25.537229 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:25.537088 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4016884f-c6vqr" event={"ID":"f40e13d6-fe4d-4ff3-b48d-4e57d1985db8","Type":"ContainerDied","Data":"c072f793d1ce993c8c67be2c80523e295bab826500c91607f26910a53ca24c9e"} Oct 30 21:53:25.537229 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:25.537184 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c072f793d1ce993c8c67be2c80523e295bab826500c91607f26910a53ca24c9e" Oct 30 21:53:25.823795 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:25.823735 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7395-4c98c5" podStartSLOduration=2.823720876 podStartE2EDuration="2.823720876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:25.591243905 +0000 UTC m=+3192.712317651" watchObservedRunningTime="2026-10-30 21:53:25.823720876 +0000 UTC m=+3192.944794623" Oct 30 21:53:26.468929 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:26.468871 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7395-4c98c5"] Oct 30 21:53:26.488928 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:26.488842 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7395-4c98c5"] Oct 30 21:53:26.551480 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:26.551445 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5" event={"ID":"78537147-7ebb-42ef-9295-b8d27a2560fc","Type":"ContainerStarted","Data":"6016ce8de4f5534f33436cad1d691852528df12dd86405f611739036e1a9baeb"} Oct 30 21:53:26.609069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:26.608983 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5" podStartSLOduration=2.608968313 podStartE2EDuration="2.608968313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:26.598919389 +0000 UTC m=+3193.719993134" watchObservedRunningTime="2026-10-30 21:53:26.608968313 +0000 UTC m=+3193.730042058" Oct 30 21:53:26.610694 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:26.610669 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7395-a10a42"] Oct 30 21:53:26.611306 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:26.611293 4926 cpu_manager.go:414] "containerMap: removing container" podUID="f40e13d6-fe4d-4ff3-b48d-4e57d1985db8" containerName="revoke-keys" Oct 30 21:53:26.611351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:26.611308 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f40e13d6-fe4d-4ff3-b48d-4e57d1985db8" containerName="revoke-keys" Oct 30 21:53:26.611351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:26.611341 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d9631bf6-2ea3-4adf-84fc-31b954c49a2e" containerName="test-curl-7395-4c98c5" Oct 30 21:53:26.611351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:26.611347 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9631bf6-2ea3-4adf-84fc-31b954c49a2e" containerName="test-curl-7395-4c98c5" Oct 30 21:53:26.611452 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:26.611431 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="f40e13d6-fe4d-4ff3-b48d-4e57d1985db8" containerName="revoke-keys" Oct 30 21:53:26.611452 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:26.611443 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9631bf6-2ea3-4adf-84fc-31b954c49a2e" containerName="test-curl-7395-4c98c5" Oct 30 21:53:26.615154 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:26.615132 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7395-a10a42" Oct 30 21:53:26.628408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:26.628377 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7395-a10a42"] Oct 30 21:53:26.734453 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:26.734415 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz5vr\" (UniqueName: \"kubernetes.io/projected/0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1-kube-api-access-mz5vr\") pod \"test-curl-7395-a10a42\" (UID: \"0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1\") " pod="opendatahub/test-curl-7395-a10a42" Oct 30 21:53:26.835767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:26.835658 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz5vr\" (UniqueName: \"kubernetes.io/projected/0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1-kube-api-access-mz5vr\") pod \"test-curl-7395-a10a42\" (UID: \"0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1\") " pod="opendatahub/test-curl-7395-a10a42" Oct 30 21:53:26.855866 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:26.855827 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz5vr\" (UniqueName: \"kubernetes.io/projected/0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1-kube-api-access-mz5vr\") pod \"test-curl-7395-a10a42\" (UID: \"0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1\") " pod="opendatahub/test-curl-7395-a10a42" Oct 30 21:53:26.953408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:26.953370 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7395-a10a42" Oct 30 21:53:27.263164 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:27.263128 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7395-a10a42"] Oct 30 21:53:27.264229 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:53:27.264185 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c6cd1a0_9991_4c2a_8ff8_996dc76ea9f1.slice/crio-abd1185d4f03870a6066e11ef2bca77c8e6a4b6b4cbbda8e55200dea7a54d638 WatchSource:0}: Error finding container abd1185d4f03870a6066e11ef2bca77c8e6a4b6b4cbbda8e55200dea7a54d638: Status 404 returned error can't find the container with id abd1185d4f03870a6066e11ef2bca77c8e6a4b6b4cbbda8e55200dea7a54d638 Oct 30 21:53:27.564076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:27.563968 4926 generic.go:356] "Generic (PLEG): container finished" podID="78537147-7ebb-42ef-9295-b8d27a2560fc" containerID="6016ce8de4f5534f33436cad1d691852528df12dd86405f611739036e1a9baeb" exitCode=0 Oct 30 21:53:27.564487 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:27.564396 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5" event={"ID":"78537147-7ebb-42ef-9295-b8d27a2560fc","Type":"ContainerDied","Data":"6016ce8de4f5534f33436cad1d691852528df12dd86405f611739036e1a9baeb"} Oct 30 21:53:27.566343 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:27.566291 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7395-4c98c5" podUID="d9631bf6-2ea3-4adf-84fc-31b954c49a2e" containerName="test-curl-7395-4c98c5" containerID="cri-o://08e57781ed778e81dbf3bcd0ed1fffd21a8618edb905b1a12d1b5e54e5922424" gracePeriod=2 Oct 30 21:53:27.567185 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:27.567155 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7395-a10a42" event={"ID":"0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1","Type":"ContainerStarted","Data":"00fba33eecbf4047c86a409117294a789c4deaa9c49935279f299a464ac0ad63"} Oct 30 21:53:27.567305 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:27.567194 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7395-a10a42" event={"ID":"0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1","Type":"ContainerStarted","Data":"abd1185d4f03870a6066e11ef2bca77c8e6a4b6b4cbbda8e55200dea7a54d638"} Oct 30 21:53:27.637705 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:27.637644 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7395-a10a42" podStartSLOduration=1.6376291520000001 podStartE2EDuration="1.637629152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:27.636978858 +0000 UTC m=+3194.758052605" watchObservedRunningTime="2026-10-30 21:53:27.637629152 +0000 UTC m=+3194.758702897" Oct 30 21:53:28.494056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:28.489977 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7395-a10a42"] Oct 30 21:53:28.513942 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:28.513900 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7395-a10a42"] Oct 30 21:53:28.755464 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:28.755391 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5" Oct 30 21:53:28.763686 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:28.763654 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78537147-7ebb-42ef-9295-b8d27a2560fc-kube-api-access-n8g6q\" (UniqueName: \"kubernetes.io/projected/78537147-7ebb-42ef-9295-b8d27a2560fc-kube-api-access-n8g6q\") pod \"78537147-7ebb-42ef-9295-b8d27a2560fc\" (UID: \"78537147-7ebb-42ef-9295-b8d27a2560fc\") " Oct 30 21:53:28.763865 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:28.763737 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78537147-7ebb-42ef-9295-b8d27a2560fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78537147-7ebb-42ef-9295-b8d27a2560fc-maas-api-xks-ca\") pod \"78537147-7ebb-42ef-9295-b8d27a2560fc\" (UID: \"78537147-7ebb-42ef-9295-b8d27a2560fc\") " Oct 30 21:53:28.764011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:28.763981 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78537147-7ebb-42ef-9295-b8d27a2560fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78537147-7ebb-42ef-9295-b8d27a2560fc-maas-api-openshift-ca\") pod \"78537147-7ebb-42ef-9295-b8d27a2560fc\" (UID: \"78537147-7ebb-42ef-9295-b8d27a2560fc\") " Oct 30 21:53:28.764462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:28.764432 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78537147-7ebb-42ef-9295-b8d27a2560fc-maas-api-openshift-ca" pod "78537147-7ebb-42ef-9295-b8d27a2560fc" (UID: "78537147-7ebb-42ef-9295-b8d27a2560fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:28.764685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:28.764648 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78537147-7ebb-42ef-9295-b8d27a2560fc-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:53:28.766369 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:28.766341 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78537147-7ebb-42ef-9295-b8d27a2560fc-maas-api-xks-ca" pod "78537147-7ebb-42ef-9295-b8d27a2560fc" (UID: "78537147-7ebb-42ef-9295-b8d27a2560fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:28.766582 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:28.766558 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78537147-7ebb-42ef-9295-b8d27a2560fc-kube-api-access-n8g6q" pod "78537147-7ebb-42ef-9295-b8d27a2560fc" (UID: "78537147-7ebb-42ef-9295-b8d27a2560fc"). InnerVolumeSpecName "kube-api-access-n8g6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:28.865853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:28.865814 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78537147-7ebb-42ef-9295-b8d27a2560fc-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:53:28.865853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:28.865849 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8g6q\" (UniqueName: \"kubernetes.io/projected/78537147-7ebb-42ef-9295-b8d27a2560fc-kube-api-access-n8g6q\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:53:28.957978 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:28.957939 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46"] Oct 30 21:53:28.958621 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:28.958601 4926 cpu_manager.go:414] "containerMap: removing container" podUID="0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1" containerName="test-curl-7395-a10a42" Oct 30 21:53:28.958621 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:28.958618 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1" containerName="test-curl-7395-a10a42" Oct 30 21:53:28.958785 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:28.958666 4926 cpu_manager.go:414] "containerMap: removing container" podUID="78537147-7ebb-42ef-9295-b8d27a2560fc" containerName="revoke-keys" Oct 30 21:53:28.958785 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:28.958672 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78537147-7ebb-42ef-9295-b8d27a2560fc" containerName="revoke-keys" Oct 30 21:53:28.958785 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:28.958775 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="78537147-7ebb-42ef-9295-b8d27a2560fc" containerName="revoke-keys" Oct 30 21:53:28.958899 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:28.958789 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1" containerName="test-curl-7395-a10a42" Oct 30 21:53:28.978315 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:28.977673 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46" Oct 30 21:53:28.993101 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:28.993001 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46"] Oct 30 21:53:29.070563 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:29.070520 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b09edca-4699-4a05-9109-d8b0b11af8bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3553cc4-ncx46\" (UID: \"1b09edca-4699-4a05-9109-d8b0b11af8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46" Oct 30 21:53:29.070788 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:29.070732 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b09edca-4699-4a05-9109-d8b0b11af8bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3553cc4-ncx46\" (UID: \"1b09edca-4699-4a05-9109-d8b0b11af8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46" Oct 30 21:53:29.070872 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:29.070795 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7hvh\" (UniqueName: \"kubernetes.io/projected/1b09edca-4699-4a05-9109-d8b0b11af8bf-kube-api-access-f7hvh\") pod \"maas-api-revoke-sub-d3553cc4-ncx46\" (UID: \"1b09edca-4699-4a05-9109-d8b0b11af8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46" Oct 30 21:53:29.173108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:29.173064 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b09edca-4699-4a05-9109-d8b0b11af8bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3553cc4-ncx46\" (UID: \"1b09edca-4699-4a05-9109-d8b0b11af8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46" Oct 30 21:53:29.173291 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:29.173227 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b09edca-4699-4a05-9109-d8b0b11af8bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3553cc4-ncx46\" (UID: \"1b09edca-4699-4a05-9109-d8b0b11af8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46" Oct 30 21:53:29.173346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:29.173295 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7hvh\" (UniqueName: \"kubernetes.io/projected/1b09edca-4699-4a05-9109-d8b0b11af8bf-kube-api-access-f7hvh\") pod \"maas-api-revoke-sub-d3553cc4-ncx46\" (UID: \"1b09edca-4699-4a05-9109-d8b0b11af8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46" Oct 30 21:53:29.174086 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:29.174025 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b09edca-4699-4a05-9109-d8b0b11af8bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3553cc4-ncx46\" (UID: \"1b09edca-4699-4a05-9109-d8b0b11af8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46" Oct 30 21:53:29.175660 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:29.175635 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b09edca-4699-4a05-9109-d8b0b11af8bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3553cc4-ncx46\" (UID: \"1b09edca-4699-4a05-9109-d8b0b11af8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46" Oct 30 21:53:29.192236 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:29.192204 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7hvh\" (UniqueName: \"kubernetes.io/projected/1b09edca-4699-4a05-9109-d8b0b11af8bf-kube-api-access-f7hvh\") pod \"maas-api-revoke-sub-d3553cc4-ncx46\" (UID: \"1b09edca-4699-4a05-9109-d8b0b11af8bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46" Oct 30 21:53:29.304096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:29.303962 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46" Oct 30 21:53:29.587024 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:29.586997 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5" Oct 30 21:53:29.587203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:29.587076 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5" event={"ID":"78537147-7ebb-42ef-9295-b8d27a2560fc","Type":"ContainerDied","Data":"9c7afeec7fbbc4bc1c5f0a0ac2ee3024877361360a591744859396f30398baf6"} Oct 30 21:53:29.587203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:29.587125 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c7afeec7fbbc4bc1c5f0a0ac2ee3024877361360a591744859396f30398baf6" Oct 30 21:53:29.587354 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:29.587272 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7395-a10a42" podUID="0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1" containerName="test-curl-7395-a10a42" containerID="cri-o://00fba33eecbf4047c86a409117294a789c4deaa9c49935279f299a464ac0ad63" gracePeriod=2 Oct 30 21:53:29.619858 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:29.619818 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46"] Oct 30 21:53:29.638674 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:53:29.638628 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b09edca_4699_4a05_9109_d8b0b11af8bf.slice/crio-b5c3d3748ed022b0224e2ac8b33c018d79c039644cb5cc3a92ff35a9aaa75742 WatchSource:0}: Error finding container b5c3d3748ed022b0224e2ac8b33c018d79c039644cb5cc3a92ff35a9aaa75742: Status 404 returned error can't find the container with id b5c3d3748ed022b0224e2ac8b33c018d79c039644cb5cc3a92ff35a9aaa75742 Oct 30 21:53:29.824812 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:29.824740 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7395-4c98c5" Oct 30 21:53:29.883549 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:29.883512 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9631bf6-2ea3-4adf-84fc-31b954c49a2e-kube-api-access-zp9s9\" (UniqueName: \"kubernetes.io/projected/d9631bf6-2ea3-4adf-84fc-31b954c49a2e-kube-api-access-zp9s9\") pod \"d9631bf6-2ea3-4adf-84fc-31b954c49a2e\" (UID: \"d9631bf6-2ea3-4adf-84fc-31b954c49a2e\") " Oct 30 21:53:29.885882 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:29.885846 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9631bf6-2ea3-4adf-84fc-31b954c49a2e-kube-api-access-zp9s9" pod "d9631bf6-2ea3-4adf-84fc-31b954c49a2e" (UID: "d9631bf6-2ea3-4adf-84fc-31b954c49a2e"). InnerVolumeSpecName "kube-api-access-zp9s9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:29.985770 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:29.985726 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp9s9\" (UniqueName: \"kubernetes.io/projected/d9631bf6-2ea3-4adf-84fc-31b954c49a2e-kube-api-access-zp9s9\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:53:30.597892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:30.597855 4926 generic.go:356] "Generic (PLEG): container finished" podID="d9631bf6-2ea3-4adf-84fc-31b954c49a2e" containerID="08e57781ed778e81dbf3bcd0ed1fffd21a8618edb905b1a12d1b5e54e5922424" exitCode=137 Oct 30 21:53:30.598177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:30.597902 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7395-4c98c5" Oct 30 21:53:30.598177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:30.597962 4926 scope.go:122] "RemoveContainer" containerID="08e57781ed778e81dbf3bcd0ed1fffd21a8618edb905b1a12d1b5e54e5922424" Oct 30 21:53:30.601097 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:30.601065 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" event={"ID":"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6","Type":"ContainerDied","Data":"2068931c23566279d1cb1eb15b62341ab40c38c9d3d0c4d36f7aefd8643d0b06"} Oct 30 21:53:30.601230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:30.601091 4926 generic.go:356] "Generic (PLEG): container finished" podID="e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" containerID="2068931c23566279d1cb1eb15b62341ab40c38c9d3d0c4d36f7aefd8643d0b06" exitCode=0 Oct 30 21:53:30.601602 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:30.601581 4926 scope.go:122] "RemoveContainer" containerID="2068931c23566279d1cb1eb15b62341ab40c38c9d3d0c4d36f7aefd8643d0b06" Oct 30 21:53:30.603615 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:30.603578 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:53:30.603750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:30.603675 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:53:30.603750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:30.603717 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:53:30.604766 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:30.604743 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46" event={"ID":"1b09edca-4699-4a05-9109-d8b0b11af8bf","Type":"ContainerStarted","Data":"253fd78f6b35d546c7b1f41cf15e27c65764a05a9ba1310fac4223bbdfd07a4a"} Oct 30 21:53:30.604861 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:30.604792 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46" event={"ID":"1b09edca-4699-4a05-9109-d8b0b11af8bf","Type":"ContainerStarted","Data":"b5c3d3748ed022b0224e2ac8b33c018d79c039644cb5cc3a92ff35a9aaa75742"} Oct 30 21:53:30.610617 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:30.610597 4926 scope.go:122] "RemoveContainer" containerID="08e57781ed778e81dbf3bcd0ed1fffd21a8618edb905b1a12d1b5e54e5922424" Oct 30 21:53:30.610921 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:53:30.610901 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08e57781ed778e81dbf3bcd0ed1fffd21a8618edb905b1a12d1b5e54e5922424\": container with ID starting with 08e57781ed778e81dbf3bcd0ed1fffd21a8618edb905b1a12d1b5e54e5922424 not found: ID does not exist" containerID="08e57781ed778e81dbf3bcd0ed1fffd21a8618edb905b1a12d1b5e54e5922424" Oct 30 21:53:30.610994 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:30.610930 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08e57781ed778e81dbf3bcd0ed1fffd21a8618edb905b1a12d1b5e54e5922424"} err="failed to get container status \"08e57781ed778e81dbf3bcd0ed1fffd21a8618edb905b1a12d1b5e54e5922424\": rpc error: code = NotFound desc = could not find container \"08e57781ed778e81dbf3bcd0ed1fffd21a8618edb905b1a12d1b5e54e5922424\": container with ID starting with 08e57781ed778e81dbf3bcd0ed1fffd21a8618edb905b1a12d1b5e54e5922424 not found: ID does not exist" Oct 30 21:53:30.715674 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:53:30.715363 4926 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7395-4c98c5\" is forbidden: User \"system:node:ip-10-0-142-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-3.ec2.internal' and this object" podUID="d9631bf6-2ea3-4adf-84fc-31b954c49a2e" pod="opendatahub/test-curl-7395-4c98c5" Oct 30 21:53:31.022387 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:31.022352 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9631bf6-2ea3-4adf-84fc-31b954c49a2e" path="/var/lib/kubelet/pods/d9631bf6-2ea3-4adf-84fc-31b954c49a2e/volumes" Oct 30 21:53:31.332121 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:31.328144 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:53:31.643155 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:31.642367 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" event={"ID":"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6","Type":"ContainerStarted","Data":"6556f37eef023f1c2c84e045cab721271550a5488205f608f502a2bb40dd1574"} Oct 30 21:53:31.644750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:31.644521 4926 generic.go:356] "Generic (PLEG): container finished" podID="1b09edca-4699-4a05-9109-d8b0b11af8bf" containerID="253fd78f6b35d546c7b1f41cf15e27c65764a05a9ba1310fac4223bbdfd07a4a" exitCode=0 Oct 30 21:53:31.644750 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:31.644598 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46" event={"ID":"1b09edca-4699-4a05-9109-d8b0b11af8bf","Type":"ContainerDied","Data":"253fd78f6b35d546c7b1f41cf15e27c65764a05a9ba1310fac4223bbdfd07a4a"} Oct 30 21:53:31.707692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:31.707622 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46" podStartSLOduration=3.707599347 podStartE2EDuration="3.707599347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:30.701893724 +0000 UTC m=+3197.822967546" watchObservedRunningTime="2026-10-30 21:53:31.707599347 +0000 UTC m=+3198.828673094" Oct 30 21:53:31.823377 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:31.823345 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:53:31.824059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:31.824012 4926 patch_prober.go:29] interesting pod/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 21:53:31.824208 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:31.824089 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" podUID="e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 21:53:31.956308 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:31.956273 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7395-a10a42" Oct 30 21:53:32.046553 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:32.046516 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1-kube-api-access-mz5vr\" (UniqueName: \"kubernetes.io/projected/0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1-kube-api-access-mz5vr\") pod \"0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1\" (UID: \"0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1\") " Oct 30 21:53:32.048885 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:32.048845 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1-kube-api-access-mz5vr" pod "0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1" (UID: "0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1"). InnerVolumeSpecName "kube-api-access-mz5vr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:32.148938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:32.148889 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz5vr\" (UniqueName: \"kubernetes.io/projected/0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1-kube-api-access-mz5vr\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:53:32.657420 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:32.657379 4926 generic.go:356] "Generic (PLEG): container finished" podID="0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1" containerID="00fba33eecbf4047c86a409117294a789c4deaa9c49935279f299a464ac0ad63" exitCode=137 Oct 30 21:53:32.657652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:32.657424 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7395-a10a42" Oct 30 21:53:32.657652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:32.657484 4926 scope.go:122] "RemoveContainer" containerID="00fba33eecbf4047c86a409117294a789c4deaa9c49935279f299a464ac0ad63" Oct 30 21:53:32.693137 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:53:32.691341 4926 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7395-a10a42\" is forbidden: User \"system:node:ip-10-0-142-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-3.ec2.internal' and this object" podUID="0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1" pod="opendatahub/test-curl-7395-a10a42" Oct 30 21:53:32.708058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:32.706225 4926 scope.go:122] "RemoveContainer" containerID="00fba33eecbf4047c86a409117294a789c4deaa9c49935279f299a464ac0ad63" Oct 30 21:53:32.708058 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:53:32.706744 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00fba33eecbf4047c86a409117294a789c4deaa9c49935279f299a464ac0ad63\": container with ID starting with 00fba33eecbf4047c86a409117294a789c4deaa9c49935279f299a464ac0ad63 not found: ID does not exist" containerID="00fba33eecbf4047c86a409117294a789c4deaa9c49935279f299a464ac0ad63" Oct 30 21:53:32.708058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:32.706789 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00fba33eecbf4047c86a409117294a789c4deaa9c49935279f299a464ac0ad63"} err="failed to get container status \"00fba33eecbf4047c86a409117294a789c4deaa9c49935279f299a464ac0ad63\": rpc error: code = NotFound desc = could not find container \"00fba33eecbf4047c86a409117294a789c4deaa9c49935279f299a464ac0ad63\": container with ID starting with 00fba33eecbf4047c86a409117294a789c4deaa9c49935279f299a464ac0ad63 not found: ID does not exist" Oct 30 21:53:32.824704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:32.824395 4926 patch_prober.go:29] interesting pod/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 21:53:32.824704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:32.824466 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" podUID="e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 21:53:32.943215 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:32.943187 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46" Oct 30 21:53:32.955917 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:53:32.955872 4926 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7395-a10a42\" is forbidden: User \"system:node:ip-10-0-142-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-3.ec2.internal' and this object" podUID="0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1" pod="opendatahub/test-curl-7395-a10a42" Oct 30 21:53:33.024104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:33.023528 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1" path="/var/lib/kubelet/pods/0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1/volumes" Oct 30 21:53:33.035763 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:53:33.035733 4926 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7395-a10a42\" is forbidden: User \"system:node:ip-10-0-142-3.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-3.ec2.internal' and this object" podUID="0c6cd1a0-9991-4c2a-8ff8-996dc76ea9f1" pod="opendatahub/test-curl-7395-a10a42" Oct 30 21:53:33.062534 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:33.062492 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b09edca-4699-4a05-9109-d8b0b11af8bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b09edca-4699-4a05-9109-d8b0b11af8bf-maas-api-openshift-ca\") pod \"1b09edca-4699-4a05-9109-d8b0b11af8bf\" (UID: \"1b09edca-4699-4a05-9109-d8b0b11af8bf\") " Oct 30 21:53:33.062945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:33.062666 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b09edca-4699-4a05-9109-d8b0b11af8bf-kube-api-access-f7hvh\" (UniqueName: \"kubernetes.io/projected/1b09edca-4699-4a05-9109-d8b0b11af8bf-kube-api-access-f7hvh\") pod \"1b09edca-4699-4a05-9109-d8b0b11af8bf\" (UID: \"1b09edca-4699-4a05-9109-d8b0b11af8bf\") " Oct 30 21:53:33.062945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:33.062791 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b09edca-4699-4a05-9109-d8b0b11af8bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b09edca-4699-4a05-9109-d8b0b11af8bf-maas-api-xks-ca\") pod \"1b09edca-4699-4a05-9109-d8b0b11af8bf\" (UID: \"1b09edca-4699-4a05-9109-d8b0b11af8bf\") " Oct 30 21:53:33.062945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:33.062803 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b09edca-4699-4a05-9109-d8b0b11af8bf-maas-api-openshift-ca" pod "1b09edca-4699-4a05-9109-d8b0b11af8bf" (UID: "1b09edca-4699-4a05-9109-d8b0b11af8bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:33.063674 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:33.063647 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b09edca-4699-4a05-9109-d8b0b11af8bf-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:53:33.065241 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:33.065216 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b09edca-4699-4a05-9109-d8b0b11af8bf-kube-api-access-f7hvh" pod "1b09edca-4699-4a05-9109-d8b0b11af8bf" (UID: "1b09edca-4699-4a05-9109-d8b0b11af8bf"). InnerVolumeSpecName "kube-api-access-f7hvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:33.103053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:33.101620 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b09edca-4699-4a05-9109-d8b0b11af8bf-maas-api-xks-ca" pod "1b09edca-4699-4a05-9109-d8b0b11af8bf" (UID: "1b09edca-4699-4a05-9109-d8b0b11af8bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:33.165123 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:33.165078 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7hvh\" (UniqueName: \"kubernetes.io/projected/1b09edca-4699-4a05-9109-d8b0b11af8bf-kube-api-access-f7hvh\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:53:33.165123 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:33.165119 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b09edca-4699-4a05-9109-d8b0b11af8bf-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:53:33.684060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:33.682129 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46" Oct 30 21:53:33.684060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:33.682277 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46" event={"ID":"1b09edca-4699-4a05-9109-d8b0b11af8bf","Type":"ContainerDied","Data":"b5c3d3748ed022b0224e2ac8b33c018d79c039644cb5cc3a92ff35a9aaa75742"} Oct 30 21:53:33.684060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:33.682328 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5c3d3748ed022b0224e2ac8b33c018d79c039644cb5cc3a92ff35a9aaa75742" Oct 30 21:53:33.749412 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:33.749374 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj"] Oct 30 21:53:33.823957 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:33.822746 4926 patch_prober.go:29] interesting pod/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 21:53:33.823957 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:33.822805 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" podUID="e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 21:53:34.823053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:34.823003 4926 patch_prober.go:29] interesting pod/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 21:53:34.823458 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:34.823081 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" podUID="e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 21:53:35.822505 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:35.822473 4926 patch_prober.go:29] interesting pod/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 21:53:35.822701 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:35.822529 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" podUID="e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 21:53:36.822892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:36.822855 4926 patch_prober.go:29] interesting pod/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body= Oct 30 21:53:36.823442 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:36.822920 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" podUID="e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" Oct 30 21:53:37.487703 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:53:37.487661 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:53:37.828346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:37.828248 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:53:37.828818 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:37.828759 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:53:37.829712 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:37.829691 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:53:38.749912 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:38.749871 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj" podUID="7b99d362-edb2-44b1-bfc7-a6688359aff5" containerName="payload-processing" containerID="cri-o://cd0e52ee095d99326a82ea0707ecd299b6722ac37d90df3cd3fc8b8058242d8e" gracePeriod=25 Oct 30 21:53:39.056573 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.056547 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj" Oct 30 21:53:39.188324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.188278 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b99d362-edb2-44b1-bfc7-a6688359aff5-kube-api-access-jbqqx\" (UniqueName: \"kubernetes.io/projected/7b99d362-edb2-44b1-bfc7-a6688359aff5-kube-api-access-jbqqx\") pod \"7b99d362-edb2-44b1-bfc7-a6688359aff5\" (UID: \"7b99d362-edb2-44b1-bfc7-a6688359aff5\") " Oct 30 21:53:39.188561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.188546 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b99d362-edb2-44b1-bfc7-a6688359aff5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b99d362-edb2-44b1-bfc7-a6688359aff5-plugins-config-volume\") pod \"7b99d362-edb2-44b1-bfc7-a6688359aff5\" (UID: \"7b99d362-edb2-44b1-bfc7-a6688359aff5\") " Oct 30 21:53:39.188944 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.188916 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b99d362-edb2-44b1-bfc7-a6688359aff5-plugins-config-volume" pod "7b99d362-edb2-44b1-bfc7-a6688359aff5" (UID: "7b99d362-edb2-44b1-bfc7-a6688359aff5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:39.189127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.189109 4926 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b99d362-edb2-44b1-bfc7-a6688359aff5-plugins-config-volume\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:53:39.190839 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.190810 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b99d362-edb2-44b1-bfc7-a6688359aff5-kube-api-access-jbqqx" pod "7b99d362-edb2-44b1-bfc7-a6688359aff5" (UID: "7b99d362-edb2-44b1-bfc7-a6688359aff5"). InnerVolumeSpecName "kube-api-access-jbqqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:39.281058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.280943 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz"] Oct 30 21:53:39.281701 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.281677 4926 cpu_manager.go:414] "containerMap: removing container" podUID="7b99d362-edb2-44b1-bfc7-a6688359aff5" containerName="payload-processing" Oct 30 21:53:39.281701 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.281701 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b99d362-edb2-44b1-bfc7-a6688359aff5" containerName="payload-processing" Oct 30 21:53:39.281892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.281747 4926 cpu_manager.go:414] "containerMap: removing container" podUID="1b09edca-4699-4a05-9109-d8b0b11af8bf" containerName="revoke-keys" Oct 30 21:53:39.281892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.281755 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b09edca-4699-4a05-9109-d8b0b11af8bf" containerName="revoke-keys" Oct 30 21:53:39.282010 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.281927 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b99d362-edb2-44b1-bfc7-a6688359aff5" containerName="payload-processing" Oct 30 21:53:39.282010 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.281947 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b09edca-4699-4a05-9109-d8b0b11af8bf" containerName="revoke-keys" Oct 30 21:53:39.284858 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.284837 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz" Oct 30 21:53:39.290308 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.290282 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbqqx\" (UniqueName: \"kubernetes.io/projected/7b99d362-edb2-44b1-bfc7-a6688359aff5-kube-api-access-jbqqx\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:53:39.303922 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.303889 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:53:39.304101 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.303945 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:53:39.317762 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.317730 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz"] Oct 30 21:53:39.392188 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.392148 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c5e81a-a031-4691-965f-1065db931bc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0472f399-drpbz\" (UID: \"b0c5e81a-a031-4691-965f-1065db931bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz" Oct 30 21:53:39.392392 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.392305 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c5e81a-a031-4691-965f-1065db931bc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0472f399-drpbz\" (UID: \"b0c5e81a-a031-4691-965f-1065db931bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz" Oct 30 21:53:39.392392 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.392353 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k59mj\" (UniqueName: \"kubernetes.io/projected/b0c5e81a-a031-4691-965f-1065db931bc4-kube-api-access-k59mj\") pod \"maas-api-revoke-sub-0472f399-drpbz\" (UID: \"b0c5e81a-a031-4691-965f-1065db931bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz" Oct 30 21:53:39.493992 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.493944 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c5e81a-a031-4691-965f-1065db931bc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0472f399-drpbz\" (UID: \"b0c5e81a-a031-4691-965f-1065db931bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz" Oct 30 21:53:39.494221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.494099 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c5e81a-a031-4691-965f-1065db931bc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0472f399-drpbz\" (UID: \"b0c5e81a-a031-4691-965f-1065db931bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz" Oct 30 21:53:39.494221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.494163 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k59mj\" (UniqueName: \"kubernetes.io/projected/b0c5e81a-a031-4691-965f-1065db931bc4-kube-api-access-k59mj\") pod \"maas-api-revoke-sub-0472f399-drpbz\" (UID: \"b0c5e81a-a031-4691-965f-1065db931bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz" Oct 30 21:53:39.494828 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.494778 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c5e81a-a031-4691-965f-1065db931bc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0472f399-drpbz\" (UID: \"b0c5e81a-a031-4691-965f-1065db931bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz" Oct 30 21:53:39.497297 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.497274 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c5e81a-a031-4691-965f-1065db931bc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0472f399-drpbz\" (UID: \"b0c5e81a-a031-4691-965f-1065db931bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz" Oct 30 21:53:39.515159 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.515130 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k59mj\" (UniqueName: \"kubernetes.io/projected/b0c5e81a-a031-4691-965f-1065db931bc4-kube-api-access-k59mj\") pod \"maas-api-revoke-sub-0472f399-drpbz\" (UID: \"b0c5e81a-a031-4691-965f-1065db931bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz" Oct 30 21:53:39.599574 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.599488 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz" Oct 30 21:53:39.757398 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.757364 4926 generic.go:356] "Generic (PLEG): container finished" podID="7b99d362-edb2-44b1-bfc7-a6688359aff5" containerID="cd0e52ee095d99326a82ea0707ecd299b6722ac37d90df3cd3fc8b8058242d8e" exitCode=0 Oct 30 21:53:39.757586 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.757409 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj" Oct 30 21:53:39.757586 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.757447 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj" event={"ID":"7b99d362-edb2-44b1-bfc7-a6688359aff5","Type":"ContainerDied","Data":"cd0e52ee095d99326a82ea0707ecd299b6722ac37d90df3cd3fc8b8058242d8e"} Oct 30 21:53:39.757586 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.757522 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj" event={"ID":"7b99d362-edb2-44b1-bfc7-a6688359aff5","Type":"ContainerDied","Data":"eb2203285f96b82237fbb1b3009236aa7e9de4b8e89ad71b3313f9a4bc231e4d"} Oct 30 21:53:39.757586 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.757559 4926 scope.go:122] "RemoveContainer" containerID="cd0e52ee095d99326a82ea0707ecd299b6722ac37d90df3cd3fc8b8058242d8e" Oct 30 21:53:39.788199 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.788170 4926 scope.go:122] "RemoveContainer" containerID="cd0e52ee095d99326a82ea0707ecd299b6722ac37d90df3cd3fc8b8058242d8e" Oct 30 21:53:39.788524 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:53:39.788500 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd0e52ee095d99326a82ea0707ecd299b6722ac37d90df3cd3fc8b8058242d8e\": container with ID starting with cd0e52ee095d99326a82ea0707ecd299b6722ac37d90df3cd3fc8b8058242d8e not found: ID does not exist" containerID="cd0e52ee095d99326a82ea0707ecd299b6722ac37d90df3cd3fc8b8058242d8e" Oct 30 21:53:39.788606 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.788533 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd0e52ee095d99326a82ea0707ecd299b6722ac37d90df3cd3fc8b8058242d8e"} err="failed to get container status \"cd0e52ee095d99326a82ea0707ecd299b6722ac37d90df3cd3fc8b8058242d8e\": rpc error: code = NotFound desc = could not find container \"cd0e52ee095d99326a82ea0707ecd299b6722ac37d90df3cd3fc8b8058242d8e\": container with ID starting with cd0e52ee095d99326a82ea0707ecd299b6722ac37d90df3cd3fc8b8058242d8e not found: ID does not exist" Oct 30 21:53:39.817566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.817528 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj"] Oct 30 21:53:39.822067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.822021 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be3ada-76ffbdb68-f8ntj"] Oct 30 21:53:39.904183 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:39.904107 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz"] Oct 30 21:53:39.908343 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:53:39.908273 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c5e81a_a031_4691_965f_1065db931bc4.slice/crio-f5ed8a53fd5ceb0ef1e68d3b1d813bdc77594db0a88ca1ea7fb9eca446f36e14 WatchSource:0}: Error finding container f5ed8a53fd5ceb0ef1e68d3b1d813bdc77594db0a88ca1ea7fb9eca446f36e14: Status 404 returned error can't find the container with id f5ed8a53fd5ceb0ef1e68d3b1d813bdc77594db0a88ca1ea7fb9eca446f36e14 Oct 30 21:53:40.767986 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:40.767948 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz" event={"ID":"b0c5e81a-a031-4691-965f-1065db931bc4","Type":"ContainerStarted","Data":"456fd2620f9a39da700c55b7151a0fd9cb9fa295b5d004bfd6c32c623a045b1d"} Oct 30 21:53:40.768463 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:40.768013 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz" event={"ID":"b0c5e81a-a031-4691-965f-1065db931bc4","Type":"ContainerStarted","Data":"f5ed8a53fd5ceb0ef1e68d3b1d813bdc77594db0a88ca1ea7fb9eca446f36e14"} Oct 30 21:53:40.808656 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:40.808597 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz" podStartSLOduration=1.8085845969999998 podStartE2EDuration="1.808584597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:40.80442305 +0000 UTC m=+3207.925496795" watchObservedRunningTime="2026-10-30 21:53:40.808584597 +0000 UTC m=+3207.929658371" Oct 30 21:53:41.019765 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:41.019681 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b99d362-edb2-44b1-bfc7-a6688359aff5" path="/var/lib/kubelet/pods/7b99d362-edb2-44b1-bfc7-a6688359aff5/volumes" Oct 30 21:53:41.783158 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:41.783119 4926 generic.go:356] "Generic (PLEG): container finished" podID="b0c5e81a-a031-4691-965f-1065db931bc4" containerID="456fd2620f9a39da700c55b7151a0fd9cb9fa295b5d004bfd6c32c623a045b1d" exitCode=0 Oct 30 21:53:41.783623 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:41.783211 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz" event={"ID":"b0c5e81a-a031-4691-965f-1065db931bc4","Type":"ContainerDied","Data":"456fd2620f9a39da700c55b7151a0fd9cb9fa295b5d004bfd6c32c623a045b1d"} Oct 30 21:53:42.990262 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:42.990233 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz" Oct 30 21:53:43.071948 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:43.071850 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0c5e81a-a031-4691-965f-1065db931bc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c5e81a-a031-4691-965f-1065db931bc4-maas-api-xks-ca\") pod \"b0c5e81a-a031-4691-965f-1065db931bc4\" (UID: \"b0c5e81a-a031-4691-965f-1065db931bc4\") " Oct 30 21:53:43.072162 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:43.072055 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0c5e81a-a031-4691-965f-1065db931bc4-kube-api-access-k59mj\" (UniqueName: \"kubernetes.io/projected/b0c5e81a-a031-4691-965f-1065db931bc4-kube-api-access-k59mj\") pod \"b0c5e81a-a031-4691-965f-1065db931bc4\" (UID: \"b0c5e81a-a031-4691-965f-1065db931bc4\") " Oct 30 21:53:43.072162 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:43.072135 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0c5e81a-a031-4691-965f-1065db931bc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c5e81a-a031-4691-965f-1065db931bc4-maas-api-openshift-ca\") pod \"b0c5e81a-a031-4691-965f-1065db931bc4\" (UID: \"b0c5e81a-a031-4691-965f-1065db931bc4\") " Oct 30 21:53:43.073179 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:43.073148 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c5e81a-a031-4691-965f-1065db931bc4-maas-api-openshift-ca" pod "b0c5e81a-a031-4691-965f-1065db931bc4" (UID: "b0c5e81a-a031-4691-965f-1065db931bc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:43.074888 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:43.074860 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c5e81a-a031-4691-965f-1065db931bc4-kube-api-access-k59mj" pod "b0c5e81a-a031-4691-965f-1065db931bc4" (UID: "b0c5e81a-a031-4691-965f-1065db931bc4"). InnerVolumeSpecName "kube-api-access-k59mj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:43.076629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:43.076602 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0c5e81a-a031-4691-965f-1065db931bc4-maas-api-xks-ca" pod "b0c5e81a-a031-4691-965f-1065db931bc4" (UID: "b0c5e81a-a031-4691-965f-1065db931bc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:43.174464 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:43.174419 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k59mj\" (UniqueName: \"kubernetes.io/projected/b0c5e81a-a031-4691-965f-1065db931bc4-kube-api-access-k59mj\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:53:43.174464 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:43.174453 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0c5e81a-a031-4691-965f-1065db931bc4-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:53:43.174464 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:43.174466 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0c5e81a-a031-4691-965f-1065db931bc4-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:53:43.805267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:43.805235 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz" Oct 30 21:53:43.805267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:43.805256 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz" event={"ID":"b0c5e81a-a031-4691-965f-1065db931bc4","Type":"ContainerDied","Data":"f5ed8a53fd5ceb0ef1e68d3b1d813bdc77594db0a88ca1ea7fb9eca446f36e14"} Oct 30 21:53:43.805505 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:43.805308 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5ed8a53fd5ceb0ef1e68d3b1d813bdc77594db0a88ca1ea7fb9eca446f36e14" Oct 30 21:53:47.833061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:47.832613 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:53:50.106594 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:50.106555 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v"] Oct 30 21:53:50.108057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:50.108008 4926 cpu_manager.go:414] "containerMap: removing container" podUID="b0c5e81a-a031-4691-965f-1065db931bc4" containerName="revoke-keys" Oct 30 21:53:50.108241 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:50.108223 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0c5e81a-a031-4691-965f-1065db931bc4" containerName="revoke-keys" Oct 30 21:53:50.108590 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:50.108569 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0c5e81a-a031-4691-965f-1065db931bc4" containerName="revoke-keys" Oct 30 21:53:50.112824 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:50.112804 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v"] Oct 30 21:53:50.112993 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:50.112924 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v" Oct 30 21:53:50.127021 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:50.126993 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:53:50.127352 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:50.127332 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:53:50.215550 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:50.215499 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bkvt\" (UniqueName: \"kubernetes.io/projected/220c70bd-bd4c-4cb4-91f3-35604adcf782-kube-api-access-2bkvt\") pod \"maas-api-revoke-sub-aad3a63f-xl24v\" (UID: \"220c70bd-bd4c-4cb4-91f3-35604adcf782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v" Oct 30 21:53:50.215775 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:50.215647 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220c70bd-bd4c-4cb4-91f3-35604adcf782-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aad3a63f-xl24v\" (UID: \"220c70bd-bd4c-4cb4-91f3-35604adcf782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v" Oct 30 21:53:50.215775 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:50.215722 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220c70bd-bd4c-4cb4-91f3-35604adcf782-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aad3a63f-xl24v\" (UID: \"220c70bd-bd4c-4cb4-91f3-35604adcf782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v" Oct 30 21:53:50.316864 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:50.316816 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bkvt\" (UniqueName: \"kubernetes.io/projected/220c70bd-bd4c-4cb4-91f3-35604adcf782-kube-api-access-2bkvt\") pod \"maas-api-revoke-sub-aad3a63f-xl24v\" (UID: \"220c70bd-bd4c-4cb4-91f3-35604adcf782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v" Oct 30 21:53:50.317065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:50.316940 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220c70bd-bd4c-4cb4-91f3-35604adcf782-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aad3a63f-xl24v\" (UID: \"220c70bd-bd4c-4cb4-91f3-35604adcf782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v" Oct 30 21:53:50.317065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:50.316977 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220c70bd-bd4c-4cb4-91f3-35604adcf782-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aad3a63f-xl24v\" (UID: \"220c70bd-bd4c-4cb4-91f3-35604adcf782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v" Oct 30 21:53:50.317835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:50.317804 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220c70bd-bd4c-4cb4-91f3-35604adcf782-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aad3a63f-xl24v\" (UID: \"220c70bd-bd4c-4cb4-91f3-35604adcf782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v" Oct 30 21:53:50.319695 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:50.319667 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220c70bd-bd4c-4cb4-91f3-35604adcf782-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aad3a63f-xl24v\" (UID: \"220c70bd-bd4c-4cb4-91f3-35604adcf782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v" Oct 30 21:53:50.337927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:50.337904 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bkvt\" (UniqueName: \"kubernetes.io/projected/220c70bd-bd4c-4cb4-91f3-35604adcf782-kube-api-access-2bkvt\") pod \"maas-api-revoke-sub-aad3a63f-xl24v\" (UID: \"220c70bd-bd4c-4cb4-91f3-35604adcf782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v" Oct 30 21:53:50.430153 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:50.430122 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v" Oct 30 21:53:50.802532 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:50.802479 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v"] Oct 30 21:53:50.804652 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:53:50.804186 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220c70bd_bd4c_4cb4_91f3_35604adcf782.slice/crio-2f53899b63b25e446eb9f4bf631d84d7555a64a6504b3630c46b58bd541fde38 WatchSource:0}: Error finding container 2f53899b63b25e446eb9f4bf631d84d7555a64a6504b3630c46b58bd541fde38: Status 404 returned error can't find the container with id 2f53899b63b25e446eb9f4bf631d84d7555a64a6504b3630c46b58bd541fde38 Oct 30 21:53:50.919105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:50.919052 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v" event={"ID":"220c70bd-bd4c-4cb4-91f3-35604adcf782","Type":"ContainerStarted","Data":"2f53899b63b25e446eb9f4bf631d84d7555a64a6504b3630c46b58bd541fde38"} Oct 30 21:53:51.929448 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:51.929399 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v" event={"ID":"220c70bd-bd4c-4cb4-91f3-35604adcf782","Type":"ContainerStarted","Data":"0d317aa4d3480ef1a500d24d0481b295deb585e8a132e63e7757674232e70b92"} Oct 30 21:53:51.968619 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:51.968556 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v" podStartSLOduration=1.968537151 podStartE2EDuration="1.968537151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:51.965570717 +0000 UTC m=+3219.086644463" watchObservedRunningTime="2026-10-30 21:53:51.968537151 +0000 UTC m=+3219.089610902" Oct 30 21:53:52.192136 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:53:52.166700 4926 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220c70bd_bd4c_4cb4_91f3_35604adcf782.slice/crio-conmon-0d317aa4d3480ef1a500d24d0481b295deb585e8a132e63e7757674232e70b92.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod220c70bd_bd4c_4cb4_91f3_35604adcf782.slice/crio-0d317aa4d3480ef1a500d24d0481b295deb585e8a132e63e7757674232e70b92.scope\": RecentStats: unable to find data in memory cache]" Oct 30 21:53:52.940352 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:52.940317 4926 generic.go:356] "Generic (PLEG): container finished" podID="220c70bd-bd4c-4cb4-91f3-35604adcf782" containerID="0d317aa4d3480ef1a500d24d0481b295deb585e8a132e63e7757674232e70b92" exitCode=0 Oct 30 21:53:52.940352 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:52.940358 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v" event={"ID":"220c70bd-bd4c-4cb4-91f3-35604adcf782","Type":"ContainerDied","Data":"0d317aa4d3480ef1a500d24d0481b295deb585e8a132e63e7757674232e70b92"} Oct 30 21:53:54.363693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:54.363664 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v" Oct 30 21:53:54.420649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:54.419463 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/220c70bd-bd4c-4cb4-91f3-35604adcf782-kube-api-access-2bkvt\" (UniqueName: \"kubernetes.io/projected/220c70bd-bd4c-4cb4-91f3-35604adcf782-kube-api-access-2bkvt\") pod \"220c70bd-bd4c-4cb4-91f3-35604adcf782\" (UID: \"220c70bd-bd4c-4cb4-91f3-35604adcf782\") " Oct 30 21:53:54.420649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:54.419546 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/220c70bd-bd4c-4cb4-91f3-35604adcf782-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220c70bd-bd4c-4cb4-91f3-35604adcf782-maas-api-openshift-ca\") pod \"220c70bd-bd4c-4cb4-91f3-35604adcf782\" (UID: \"220c70bd-bd4c-4cb4-91f3-35604adcf782\") " Oct 30 21:53:54.420649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:54.419714 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/220c70bd-bd4c-4cb4-91f3-35604adcf782-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220c70bd-bd4c-4cb4-91f3-35604adcf782-maas-api-xks-ca\") pod \"220c70bd-bd4c-4cb4-91f3-35604adcf782\" (UID: \"220c70bd-bd4c-4cb4-91f3-35604adcf782\") " Oct 30 21:53:54.421991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:54.421962 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/220c70bd-bd4c-4cb4-91f3-35604adcf782-maas-api-openshift-ca" pod "220c70bd-bd4c-4cb4-91f3-35604adcf782" (UID: "220c70bd-bd4c-4cb4-91f3-35604adcf782"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:53:54.423236 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:54.423200 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/220c70bd-bd4c-4cb4-91f3-35604adcf782-maas-api-xks-ca" pod "220c70bd-bd4c-4cb4-91f3-35604adcf782" (UID: "220c70bd-bd4c-4cb4-91f3-35604adcf782"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:53:54.424400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:54.424363 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/220c70bd-bd4c-4cb4-91f3-35604adcf782-kube-api-access-2bkvt" pod "220c70bd-bd4c-4cb4-91f3-35604adcf782" (UID: "220c70bd-bd4c-4cb4-91f3-35604adcf782"). InnerVolumeSpecName "kube-api-access-2bkvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:53:54.522000 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:54.521907 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/220c70bd-bd4c-4cb4-91f3-35604adcf782-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:53:54.522000 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:54.521947 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/220c70bd-bd4c-4cb4-91f3-35604adcf782-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:53:54.522000 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:54.521963 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bkvt\" (UniqueName: \"kubernetes.io/projected/220c70bd-bd4c-4cb4-91f3-35604adcf782-kube-api-access-2bkvt\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:53:54.958988 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:54.958947 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v" Oct 30 21:53:54.959204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:54.958984 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v" event={"ID":"220c70bd-bd4c-4cb4-91f3-35604adcf782","Type":"ContainerDied","Data":"2f53899b63b25e446eb9f4bf631d84d7555a64a6504b3630c46b58bd541fde38"} Oct 30 21:53:54.959204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:54.959098 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f53899b63b25e446eb9f4bf631d84d7555a64a6504b3630c46b58bd541fde38" Oct 30 21:53:56.289113 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:56.289072 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf"] Oct 30 21:53:56.302559 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:56.302527 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f9a8ed-f9wkf"] Oct 30 21:53:57.021410 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:57.021368 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b852d25-3186-476d-90be-f1732a4d85e2" path="/var/lib/kubelet/pods/4b852d25-3186-476d-90be-f1732a4d85e2/volumes" Oct 30 21:53:57.408280 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:53:57.408188 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq"] Oct 30 21:54:00.010450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.010413 4926 generic.go:356] "Generic (PLEG): container finished" podID="79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" containerID="db7bc3b531fa7b313086aaf969b710e52a1e7b3b6ece5c8f378e2767ccef200e" exitCode=0 Oct 30 21:54:00.011050 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.010484 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" event={"ID":"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee","Type":"ContainerDied","Data":"db7bc3b531fa7b313086aaf969b710e52a1e7b3b6ece5c8f378e2767ccef200e"} Oct 30 21:54:00.011050 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.010942 4926 scope.go:122] "RemoveContainer" containerID="db7bc3b531fa7b313086aaf969b710e52a1e7b3b6ece5c8f378e2767ccef200e" Oct 30 21:54:00.012876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.012841 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:54:00.012978 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.012934 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:54:00.012978 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.012970 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:54:00.569182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.569147 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92"] Oct 30 21:54:00.569774 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.569754 4926 cpu_manager.go:414] "containerMap: removing container" podUID="220c70bd-bd4c-4cb4-91f3-35604adcf782" containerName="revoke-keys" Oct 30 21:54:00.569774 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.569772 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="220c70bd-bd4c-4cb4-91f3-35604adcf782" containerName="revoke-keys" Oct 30 21:54:00.569916 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.569904 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="220c70bd-bd4c-4cb4-91f3-35604adcf782" containerName="revoke-keys" Oct 30 21:54:00.573720 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.573702 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92" Oct 30 21:54:00.587883 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.587842 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:00.588249 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.588219 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:54:00.594945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.594915 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92"] Oct 30 21:54:00.736002 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.735965 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4792d7f-c239-4922-a6a5-82192ca0a807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb88ee49-pkv92\" (UID: \"d4792d7f-c239-4922-a6a5-82192ca0a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92" Oct 30 21:54:00.736002 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.736014 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwtr4\" (UniqueName: \"kubernetes.io/projected/d4792d7f-c239-4922-a6a5-82192ca0a807-kube-api-access-jwtr4\") pod \"maas-api-revoke-sub-bb88ee49-pkv92\" (UID: \"d4792d7f-c239-4922-a6a5-82192ca0a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92" Oct 30 21:54:00.736273 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.736201 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4792d7f-c239-4922-a6a5-82192ca0a807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb88ee49-pkv92\" (UID: \"d4792d7f-c239-4922-a6a5-82192ca0a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92" Oct 30 21:54:00.838159 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.838066 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4792d7f-c239-4922-a6a5-82192ca0a807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb88ee49-pkv92\" (UID: \"d4792d7f-c239-4922-a6a5-82192ca0a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92" Oct 30 21:54:00.838159 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.838128 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwtr4\" (UniqueName: \"kubernetes.io/projected/d4792d7f-c239-4922-a6a5-82192ca0a807-kube-api-access-jwtr4\") pod \"maas-api-revoke-sub-bb88ee49-pkv92\" (UID: \"d4792d7f-c239-4922-a6a5-82192ca0a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92" Oct 30 21:54:00.838423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.838235 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4792d7f-c239-4922-a6a5-82192ca0a807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb88ee49-pkv92\" (UID: \"d4792d7f-c239-4922-a6a5-82192ca0a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92" Oct 30 21:54:00.839408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.839373 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4792d7f-c239-4922-a6a5-82192ca0a807-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb88ee49-pkv92\" (UID: \"d4792d7f-c239-4922-a6a5-82192ca0a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92" Oct 30 21:54:00.841214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.841158 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4792d7f-c239-4922-a6a5-82192ca0a807-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb88ee49-pkv92\" (UID: \"d4792d7f-c239-4922-a6a5-82192ca0a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92" Oct 30 21:54:00.860733 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.860695 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwtr4\" (UniqueName: \"kubernetes.io/projected/d4792d7f-c239-4922-a6a5-82192ca0a807-kube-api-access-jwtr4\") pod \"maas-api-revoke-sub-bb88ee49-pkv92\" (UID: \"d4792d7f-c239-4922-a6a5-82192ca0a807\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92" Oct 30 21:54:00.898058 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:00.897271 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92" Oct 30 21:54:01.030516 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:01.030473 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" event={"ID":"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee","Type":"ContainerStarted","Data":"1eb578b28777ec451fbac64f1b29a08aa0c065878298893470a9ac484bca32ae"} Oct 30 21:54:01.245625 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:01.245589 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92"] Oct 30 21:54:01.246320 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:54:01.246290 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4792d7f_c239_4922_a6a5_82192ca0a807.slice/crio-4d2d92a71c193284b0904a101bb5760215cd8a577ac5267ca02bd6b1dff1f60f WatchSource:0}: Error finding container 4d2d92a71c193284b0904a101bb5760215cd8a577ac5267ca02bd6b1dff1f60f: Status 404 returned error can't find the container with id 4d2d92a71c193284b0904a101bb5760215cd8a577ac5267ca02bd6b1dff1f60f Oct 30 21:54:01.407478 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:01.407440 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:54:01.407743 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:01.407664 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body= Oct 30 21:54:01.407812 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:01.407732 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" podUID="79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" Oct 30 21:54:02.040293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:02.040247 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92" event={"ID":"d4792d7f-c239-4922-a6a5-82192ca0a807","Type":"ContainerStarted","Data":"adfa268de1b2df33080d71b9d147a6c778d979339951bfab5e8fced342e8e409"} Oct 30 21:54:02.040770 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:02.040317 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92" event={"ID":"d4792d7f-c239-4922-a6a5-82192ca0a807","Type":"ContainerStarted","Data":"4d2d92a71c193284b0904a101bb5760215cd8a577ac5267ca02bd6b1dff1f60f"} Oct 30 21:54:02.085054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:02.081556 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92" podStartSLOduration=2.08153381 podStartE2EDuration="2.08153381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:02.078122134 +0000 UTC m=+3229.199195884" watchObservedRunningTime="2026-10-30 21:54:02.08153381 +0000 UTC m=+3229.202607558" Oct 30 21:54:02.407641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:02.407556 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body= Oct 30 21:54:02.407912 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:02.407876 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" podUID="79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" Oct 30 21:54:02.408641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:02.408586 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq" podUID="ba871578-1186-44c3-842d-5cfc5baa7f35" containerName="payload-processing" containerID="cri-o://d9521d8beacf8f020119c7081cf59ccaf04f04689b512e13ae13619162531999" gracePeriod=25 Oct 30 21:54:03.178055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:03.167495 4926 generic.go:356] "Generic (PLEG): container finished" podID="ba871578-1186-44c3-842d-5cfc5baa7f35" containerID="d9521d8beacf8f020119c7081cf59ccaf04f04689b512e13ae13619162531999" exitCode=0 Oct 30 21:54:03.178055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:03.167697 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq" event={"ID":"ba871578-1186-44c3-842d-5cfc5baa7f35","Type":"ContainerDied","Data":"d9521d8beacf8f020119c7081cf59ccaf04f04689b512e13ae13619162531999"} Oct 30 21:54:03.219105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:03.211490 4926 generic.go:356] "Generic (PLEG): container finished" podID="d4792d7f-c239-4922-a6a5-82192ca0a807" containerID="adfa268de1b2df33080d71b9d147a6c778d979339951bfab5e8fced342e8e409" exitCode=0 Oct 30 21:54:03.219105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:03.211607 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92" event={"ID":"d4792d7f-c239-4922-a6a5-82192ca0a807","Type":"ContainerDied","Data":"adfa268de1b2df33080d71b9d147a6c778d979339951bfab5e8fced342e8e409"} Oct 30 21:54:03.242363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:03.242332 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq" Oct 30 21:54:03.247479 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:03.247448 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n"] Oct 30 21:54:03.255875 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:03.255847 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-203209-6n95n"] Oct 30 21:54:03.265197 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:03.265168 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba871578-1186-44c3-842d-5cfc5baa7f35-kube-api-access-qc55g\" (UniqueName: \"kubernetes.io/projected/ba871578-1186-44c3-842d-5cfc5baa7f35-kube-api-access-qc55g\") pod \"ba871578-1186-44c3-842d-5cfc5baa7f35\" (UID: \"ba871578-1186-44c3-842d-5cfc5baa7f35\") " Oct 30 21:54:03.265334 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:03.265271 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba871578-1186-44c3-842d-5cfc5baa7f35-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba871578-1186-44c3-842d-5cfc5baa7f35-plugins-config-volume\") pod \"ba871578-1186-44c3-842d-5cfc5baa7f35\" (UID: \"ba871578-1186-44c3-842d-5cfc5baa7f35\") " Oct 30 21:54:03.265679 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:03.265650 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba871578-1186-44c3-842d-5cfc5baa7f35-plugins-config-volume" pod "ba871578-1186-44c3-842d-5cfc5baa7f35" (UID: "ba871578-1186-44c3-842d-5cfc5baa7f35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:03.265922 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:03.265900 4926 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba871578-1186-44c3-842d-5cfc5baa7f35-plugins-config-volume\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:03.305923 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:03.305859 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba871578-1186-44c3-842d-5cfc5baa7f35-kube-api-access-qc55g" pod "ba871578-1186-44c3-842d-5cfc5baa7f35" (UID: "ba871578-1186-44c3-842d-5cfc5baa7f35"). InnerVolumeSpecName "kube-api-access-qc55g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:03.404827 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:03.404789 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc55g\" (UniqueName: \"kubernetes.io/projected/ba871578-1186-44c3-842d-5cfc5baa7f35-kube-api-access-qc55g\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:03.407395 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:03.407360 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body= Oct 30 21:54:03.407629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:03.407419 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" podUID="79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" Oct 30 21:54:04.289064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.282863 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq" Oct 30 21:54:04.289064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.288429 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq" event={"ID":"ba871578-1186-44c3-842d-5cfc5baa7f35","Type":"ContainerDied","Data":"c94629ee32d31150b44e7ef5f36e2d5f47ec4702bda7c3d7bbb1cc724acc840e"} Oct 30 21:54:04.289064 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.288506 4926 scope.go:122] "RemoveContainer" containerID="d9521d8beacf8f020119c7081cf59ccaf04f04689b512e13ae13619162531999" Oct 30 21:54:04.418074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.413731 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body= Oct 30 21:54:04.418074 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.415411 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" podUID="79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" Oct 30 21:54:04.420476 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.420443 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq"] Oct 30 21:54:04.428219 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.428081 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f8137c-77958df654-xxgsq"] Oct 30 21:54:04.596067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.593460 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92" Oct 30 21:54:04.658635 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.658594 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4792d7f-c239-4922-a6a5-82192ca0a807-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4792d7f-c239-4922-a6a5-82192ca0a807-maas-api-xks-ca\") pod \"d4792d7f-c239-4922-a6a5-82192ca0a807\" (UID: \"d4792d7f-c239-4922-a6a5-82192ca0a807\") " Oct 30 21:54:04.658858 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.658661 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4792d7f-c239-4922-a6a5-82192ca0a807-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4792d7f-c239-4922-a6a5-82192ca0a807-maas-api-openshift-ca\") pod \"d4792d7f-c239-4922-a6a5-82192ca0a807\" (UID: \"d4792d7f-c239-4922-a6a5-82192ca0a807\") " Oct 30 21:54:04.658858 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.658824 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4792d7f-c239-4922-a6a5-82192ca0a807-kube-api-access-jwtr4\" (UniqueName: \"kubernetes.io/projected/d4792d7f-c239-4922-a6a5-82192ca0a807-kube-api-access-jwtr4\") pod \"d4792d7f-c239-4922-a6a5-82192ca0a807\" (UID: \"d4792d7f-c239-4922-a6a5-82192ca0a807\") " Oct 30 21:54:04.659109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.659077 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4792d7f-c239-4922-a6a5-82192ca0a807-maas-api-openshift-ca" pod "d4792d7f-c239-4922-a6a5-82192ca0a807" (UID: "d4792d7f-c239-4922-a6a5-82192ca0a807"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:04.659599 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.659575 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4792d7f-c239-4922-a6a5-82192ca0a807-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:04.661081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.661054 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4792d7f-c239-4922-a6a5-82192ca0a807-kube-api-access-jwtr4" pod "d4792d7f-c239-4922-a6a5-82192ca0a807" (UID: "d4792d7f-c239-4922-a6a5-82192ca0a807"). InnerVolumeSpecName "kube-api-access-jwtr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:04.661226 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.661094 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4792d7f-c239-4922-a6a5-82192ca0a807-maas-api-xks-ca" pod "d4792d7f-c239-4922-a6a5-82192ca0a807" (UID: "d4792d7f-c239-4922-a6a5-82192ca0a807"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:04.729501 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.729462 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5"] Oct 30 21:54:04.730221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.730196 4926 cpu_manager.go:414] "containerMap: removing container" podUID="ba871578-1186-44c3-842d-5cfc5baa7f35" containerName="payload-processing" Oct 30 21:54:04.730221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.730222 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba871578-1186-44c3-842d-5cfc5baa7f35" containerName="payload-processing" Oct 30 21:54:04.730408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.730288 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d4792d7f-c239-4922-a6a5-82192ca0a807" containerName="revoke-keys" Oct 30 21:54:04.730408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.730297 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4792d7f-c239-4922-a6a5-82192ca0a807" containerName="revoke-keys" Oct 30 21:54:04.730514 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.730461 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4792d7f-c239-4922-a6a5-82192ca0a807" containerName="revoke-keys" Oct 30 21:54:04.730514 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.730483 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba871578-1186-44c3-842d-5cfc5baa7f35" containerName="payload-processing" Oct 30 21:54:04.734177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.734152 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5" Oct 30 21:54:04.736371 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.736341 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5"] Oct 30 21:54:04.760571 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.760540 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4792d7f-c239-4922-a6a5-82192ca0a807-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:04.760571 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.760570 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwtr4\" (UniqueName: \"kubernetes.io/projected/d4792d7f-c239-4922-a6a5-82192ca0a807-kube-api-access-jwtr4\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:04.861858 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.861759 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjzpn\" (UniqueName: \"kubernetes.io/projected/aed9789d-3a21-41f9-971b-d353d25f9ebc-kube-api-access-zjzpn\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5\" (UID: \"aed9789d-3a21-41f9-971b-d353d25f9ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5" Oct 30 21:54:04.862022 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.861958 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed9789d-3a21-41f9-971b-d353d25f9ebc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5\" (UID: \"aed9789d-3a21-41f9-971b-d353d25f9ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5" Oct 30 21:54:04.862149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.862133 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed9789d-3a21-41f9-971b-d353d25f9ebc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5\" (UID: \"aed9789d-3a21-41f9-971b-d353d25f9ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5" Oct 30 21:54:04.963454 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.963416 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed9789d-3a21-41f9-971b-d353d25f9ebc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5\" (UID: \"aed9789d-3a21-41f9-971b-d353d25f9ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5" Oct 30 21:54:04.963664 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.963481 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjzpn\" (UniqueName: \"kubernetes.io/projected/aed9789d-3a21-41f9-971b-d353d25f9ebc-kube-api-access-zjzpn\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5\" (UID: \"aed9789d-3a21-41f9-971b-d353d25f9ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5" Oct 30 21:54:04.963664 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.963619 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed9789d-3a21-41f9-971b-d353d25f9ebc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5\" (UID: \"aed9789d-3a21-41f9-971b-d353d25f9ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5" Oct 30 21:54:04.964971 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.964136 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed9789d-3a21-41f9-971b-d353d25f9ebc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5\" (UID: \"aed9789d-3a21-41f9-971b-d353d25f9ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5" Oct 30 21:54:04.977104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.966095 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed9789d-3a21-41f9-971b-d353d25f9ebc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5\" (UID: \"aed9789d-3a21-41f9-971b-d353d25f9ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5" Oct 30 21:54:05.002106 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:04.992785 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjzpn\" (UniqueName: \"kubernetes.io/projected/aed9789d-3a21-41f9-971b-d353d25f9ebc-kube-api-access-zjzpn\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5\" (UID: \"aed9789d-3a21-41f9-971b-d353d25f9ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5" Oct 30 21:54:05.017179 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:05.017145 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99ee8ed4-52da-4fcb-aa6d-bb16e6765985" path="/var/lib/kubelet/pods/99ee8ed4-52da-4fcb-aa6d-bb16e6765985/volumes" Oct 30 21:54:05.017643 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:05.017628 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba871578-1186-44c3-842d-5cfc5baa7f35" path="/var/lib/kubelet/pods/ba871578-1186-44c3-842d-5cfc5baa7f35/volumes" Oct 30 21:54:05.054686 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:05.054655 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5" Oct 30 21:54:05.312374 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:05.312327 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92" event={"ID":"d4792d7f-c239-4922-a6a5-82192ca0a807","Type":"ContainerDied","Data":"4d2d92a71c193284b0904a101bb5760215cd8a577ac5267ca02bd6b1dff1f60f"} Oct 30 21:54:05.313019 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:05.312990 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92" Oct 30 21:54:05.313295 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:05.312990 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d2d92a71c193284b0904a101bb5760215cd8a577ac5267ca02bd6b1dff1f60f" Oct 30 21:54:05.407131 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:05.407096 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body= Oct 30 21:54:05.407305 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:05.407155 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" podUID="79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" Oct 30 21:54:05.461106 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:05.461073 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5"] Oct 30 21:54:05.461508 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:54:05.461480 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed9789d_3a21_41f9_971b_d353d25f9ebc.slice/crio-394893ae86c58907b220e67ea5a4c60018a61fadbcaf060525f111622e44681f WatchSource:0}: Error finding container 394893ae86c58907b220e67ea5a4c60018a61fadbcaf060525f111622e44681f: Status 404 returned error can't find the container with id 394893ae86c58907b220e67ea5a4c60018a61fadbcaf060525f111622e44681f Oct 30 21:54:06.321930 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:06.321845 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5" event={"ID":"aed9789d-3a21-41f9-971b-d353d25f9ebc","Type":"ContainerStarted","Data":"940130f85cba10400d86ddd10c173fba667a8044819259dab26a2192adb02767"} Oct 30 21:54:06.321930 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:06.321900 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5" event={"ID":"aed9789d-3a21-41f9-971b-d353d25f9ebc","Type":"ContainerStarted","Data":"394893ae86c58907b220e67ea5a4c60018a61fadbcaf060525f111622e44681f"} Oct 30 21:54:06.363713 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:06.363659 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5" podStartSLOduration=2.36364179 podStartE2EDuration="2.36364179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:06.362142024 +0000 UTC m=+3233.483215774" watchObservedRunningTime="2026-10-30 21:54:06.36364179 +0000 UTC m=+3233.484715536" Oct 30 21:54:06.407550 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:06.407509 4926 patch_prober.go:29] interesting pod/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" start-of-body= Oct 30 21:54:06.407727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:06.407566 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" podUID="79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.23:15021/healthz/ready\": dial tcp 10.132.1.23:15021: connect: connection refused" Oct 30 21:54:07.332196 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:07.332152 4926 generic.go:356] "Generic (PLEG): container finished" podID="aed9789d-3a21-41f9-971b-d353d25f9ebc" containerID="940130f85cba10400d86ddd10c173fba667a8044819259dab26a2192adb02767" exitCode=0 Oct 30 21:54:07.332644 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:07.332205 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5" event={"ID":"aed9789d-3a21-41f9-971b-d353d25f9ebc","Type":"ContainerDied","Data":"940130f85cba10400d86ddd10c173fba667a8044819259dab26a2192adb02767"} Oct 30 21:54:07.413125 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:07.413082 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:54:07.413597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:07.413568 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:54:07.414490 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:07.414438 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:54:07.485750 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:54:07.485712 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:54:08.599098 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:08.598771 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5"] Oct 30 21:54:08.605214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:08.605178 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8964ac-zdlt5"] Oct 30 21:54:08.694722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:08.694693 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5" Oct 30 21:54:08.810428 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:08.810333 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aed9789d-3a21-41f9-971b-d353d25f9ebc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed9789d-3a21-41f9-971b-d353d25f9ebc-maas-api-openshift-ca\") pod \"aed9789d-3a21-41f9-971b-d353d25f9ebc\" (UID: \"aed9789d-3a21-41f9-971b-d353d25f9ebc\") " Oct 30 21:54:08.810695 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:08.810672 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed9789d-3a21-41f9-971b-d353d25f9ebc-kube-api-access-zjzpn\" (UniqueName: \"kubernetes.io/projected/aed9789d-3a21-41f9-971b-d353d25f9ebc-kube-api-access-zjzpn\") pod \"aed9789d-3a21-41f9-971b-d353d25f9ebc\" (UID: \"aed9789d-3a21-41f9-971b-d353d25f9ebc\") " Oct 30 21:54:08.810795 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:08.810688 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed9789d-3a21-41f9-971b-d353d25f9ebc-maas-api-openshift-ca" pod "aed9789d-3a21-41f9-971b-d353d25f9ebc" (UID: "aed9789d-3a21-41f9-971b-d353d25f9ebc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:08.810795 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:08.810719 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aed9789d-3a21-41f9-971b-d353d25f9ebc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed9789d-3a21-41f9-971b-d353d25f9ebc-maas-api-xks-ca\") pod \"aed9789d-3a21-41f9-971b-d353d25f9ebc\" (UID: \"aed9789d-3a21-41f9-971b-d353d25f9ebc\") " Oct 30 21:54:08.811444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:08.811416 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed9789d-3a21-41f9-971b-d353d25f9ebc-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:08.812950 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:08.812928 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aed9789d-3a21-41f9-971b-d353d25f9ebc-maas-api-xks-ca" pod "aed9789d-3a21-41f9-971b-d353d25f9ebc" (UID: "aed9789d-3a21-41f9-971b-d353d25f9ebc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:08.813071 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:08.812996 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed9789d-3a21-41f9-971b-d353d25f9ebc-kube-api-access-zjzpn" pod "aed9789d-3a21-41f9-971b-d353d25f9ebc" (UID: "aed9789d-3a21-41f9-971b-d353d25f9ebc"). InnerVolumeSpecName "kube-api-access-zjzpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:08.912802 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:08.912759 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjzpn\" (UniqueName: \"kubernetes.io/projected/aed9789d-3a21-41f9-971b-d353d25f9ebc-kube-api-access-zjzpn\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:08.912802 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:08.912801 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed9789d-3a21-41f9-971b-d353d25f9ebc-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:09.020100 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:09.020065 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78537147-7ebb-42ef-9295-b8d27a2560fc" path="/var/lib/kubelet/pods/78537147-7ebb-42ef-9295-b8d27a2560fc/volumes" Oct 30 21:54:09.453696 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:09.453612 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5" event={"ID":"aed9789d-3a21-41f9-971b-d353d25f9ebc","Type":"ContainerDied","Data":"394893ae86c58907b220e67ea5a4c60018a61fadbcaf060525f111622e44681f"} Oct 30 21:54:09.453939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:09.453705 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="394893ae86c58907b220e67ea5a4c60018a61fadbcaf060525f111622e44681f" Oct 30 21:54:09.454106 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:09.454070 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5" Oct 30 21:54:10.032588 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:10.032550 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02d82ca1-kzlm7"] Oct 30 21:54:10.041212 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:10.041180 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02d82ca1-kzlm7"] Oct 30 21:54:11.017633 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:11.017595 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70dc8202-7b65-4ef5-b096-b5c71680ea27" path="/var/lib/kubelet/pods/70dc8202-7b65-4ef5-b096-b5c71680ea27/volumes" Oct 30 21:54:16.250536 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:16.250495 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4"] Oct 30 21:54:16.251137 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:16.251121 4926 cpu_manager.go:414] "containerMap: removing container" podUID="aed9789d-3a21-41f9-971b-d353d25f9ebc" containerName="revoke-keys" Oct 30 21:54:16.251184 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:16.251138 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aed9789d-3a21-41f9-971b-d353d25f9ebc" containerName="revoke-keys" Oct 30 21:54:16.251272 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:16.251262 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="aed9789d-3a21-41f9-971b-d353d25f9ebc" containerName="revoke-keys" Oct 30 21:54:16.255075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:16.255020 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4" Oct 30 21:54:16.274334 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:16.274299 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4"] Oct 30 21:54:16.334398 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:16.334353 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt8n6\" (UniqueName: \"kubernetes.io/projected/89f7e1d5-2080-4493-81d5-e8616335eaf9-kube-api-access-jt8n6\") pod \"maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4\" (UID: \"89f7e1d5-2080-4493-81d5-e8616335eaf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4" Oct 30 21:54:16.334593 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:16.334540 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89f7e1d5-2080-4493-81d5-e8616335eaf9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4\" (UID: \"89f7e1d5-2080-4493-81d5-e8616335eaf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4" Oct 30 21:54:16.334756 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:16.334737 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89f7e1d5-2080-4493-81d5-e8616335eaf9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4\" (UID: \"89f7e1d5-2080-4493-81d5-e8616335eaf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4" Oct 30 21:54:16.364229 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:16.364193 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:16.364380 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:16.364237 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:54:16.436528 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:16.436482 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89f7e1d5-2080-4493-81d5-e8616335eaf9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4\" (UID: \"89f7e1d5-2080-4493-81d5-e8616335eaf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4" Oct 30 21:54:16.436730 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:16.436562 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt8n6\" (UniqueName: \"kubernetes.io/projected/89f7e1d5-2080-4493-81d5-e8616335eaf9-kube-api-access-jt8n6\") pod \"maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4\" (UID: \"89f7e1d5-2080-4493-81d5-e8616335eaf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4" Oct 30 21:54:16.436730 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:16.436709 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89f7e1d5-2080-4493-81d5-e8616335eaf9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4\" (UID: \"89f7e1d5-2080-4493-81d5-e8616335eaf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4" Oct 30 21:54:16.437320 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:16.437282 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89f7e1d5-2080-4493-81d5-e8616335eaf9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4\" (UID: \"89f7e1d5-2080-4493-81d5-e8616335eaf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4" Oct 30 21:54:16.439337 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:16.439312 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89f7e1d5-2080-4493-81d5-e8616335eaf9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4\" (UID: \"89f7e1d5-2080-4493-81d5-e8616335eaf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4" Oct 30 21:54:16.456136 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:16.456101 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt8n6\" (UniqueName: \"kubernetes.io/projected/89f7e1d5-2080-4493-81d5-e8616335eaf9-kube-api-access-jt8n6\") pod \"maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4\" (UID: \"89f7e1d5-2080-4493-81d5-e8616335eaf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4" Oct 30 21:54:16.569188 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:16.569103 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4" Oct 30 21:54:16.882225 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:16.882185 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4"] Oct 30 21:54:17.561470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:17.561366 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4" event={"ID":"89f7e1d5-2080-4493-81d5-e8616335eaf9","Type":"ContainerStarted","Data":"c258d4faa910dde3eb3a9622bb8b1c8e7a0f5bd94887413dae9840542234c725"} Oct 30 21:54:17.561881 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:17.561471 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4" event={"ID":"89f7e1d5-2080-4493-81d5-e8616335eaf9","Type":"ContainerStarted","Data":"5e54070b14933265edc78118d1b11200df73933f8a2d63e281ab33f945a23e29"} Oct 30 21:54:17.600118 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:17.600014 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4" podStartSLOduration=1.599998862 podStartE2EDuration="1.599998862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:17.598342145 +0000 UTC m=+3244.719415892" watchObservedRunningTime="2026-10-30 21:54:17.599998862 +0000 UTC m=+3244.721072611" Oct 30 21:54:18.574937 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:18.574898 4926 generic.go:356] "Generic (PLEG): container finished" podID="89f7e1d5-2080-4493-81d5-e8616335eaf9" containerID="c258d4faa910dde3eb3a9622bb8b1c8e7a0f5bd94887413dae9840542234c725" exitCode=0 Oct 30 21:54:18.575373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:18.574982 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4" event={"ID":"89f7e1d5-2080-4493-81d5-e8616335eaf9","Type":"ContainerDied","Data":"c258d4faa910dde3eb3a9622bb8b1c8e7a0f5bd94887413dae9840542234c725"} Oct 30 21:54:18.848084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:18.847977 4926 scope.go:122] "RemoveContainer" containerID="5bbc76c0fa41aeb4206546529ecec98a331fc77f348c8e7603b0dd590082f778" Oct 30 21:54:18.859818 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:18.859793 4926 scope.go:122] "RemoveContainer" containerID="9349c3586c742fe2b4d934c14df798f951c97184cbd85124996bc58ba78dcf08" Oct 30 21:54:18.873444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:18.873419 4926 scope.go:122] "RemoveContainer" containerID="2d032e663517763c305fddbb3a5ecdc7c0f1c09e844ff96a29be77de55fc4a1f" Oct 30 21:54:19.764584 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:19.764554 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4" Oct 30 21:54:19.884589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:19.884494 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89f7e1d5-2080-4493-81d5-e8616335eaf9-kube-api-access-jt8n6\" (UniqueName: \"kubernetes.io/projected/89f7e1d5-2080-4493-81d5-e8616335eaf9-kube-api-access-jt8n6\") pod \"89f7e1d5-2080-4493-81d5-e8616335eaf9\" (UID: \"89f7e1d5-2080-4493-81d5-e8616335eaf9\") " Oct 30 21:54:19.884589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:19.884538 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89f7e1d5-2080-4493-81d5-e8616335eaf9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89f7e1d5-2080-4493-81d5-e8616335eaf9-maas-api-openshift-ca\") pod \"89f7e1d5-2080-4493-81d5-e8616335eaf9\" (UID: \"89f7e1d5-2080-4493-81d5-e8616335eaf9\") " Oct 30 21:54:19.884850 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:19.884721 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89f7e1d5-2080-4493-81d5-e8616335eaf9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89f7e1d5-2080-4493-81d5-e8616335eaf9-maas-api-xks-ca\") pod \"89f7e1d5-2080-4493-81d5-e8616335eaf9\" (UID: \"89f7e1d5-2080-4493-81d5-e8616335eaf9\") " Oct 30 21:54:19.885024 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:19.884993 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89f7e1d5-2080-4493-81d5-e8616335eaf9-maas-api-openshift-ca" pod "89f7e1d5-2080-4493-81d5-e8616335eaf9" (UID: "89f7e1d5-2080-4493-81d5-e8616335eaf9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:19.885176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:19.885154 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89f7e1d5-2080-4493-81d5-e8616335eaf9-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:19.886820 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:19.886758 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89f7e1d5-2080-4493-81d5-e8616335eaf9-maas-api-xks-ca" pod "89f7e1d5-2080-4493-81d5-e8616335eaf9" (UID: "89f7e1d5-2080-4493-81d5-e8616335eaf9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:19.887220 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:19.887191 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f7e1d5-2080-4493-81d5-e8616335eaf9-kube-api-access-jt8n6" pod "89f7e1d5-2080-4493-81d5-e8616335eaf9" (UID: "89f7e1d5-2080-4493-81d5-e8616335eaf9"). InnerVolumeSpecName "kube-api-access-jt8n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:19.987088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:19.987021 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt8n6\" (UniqueName: \"kubernetes.io/projected/89f7e1d5-2080-4493-81d5-e8616335eaf9-kube-api-access-jt8n6\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:19.987088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:19.987086 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89f7e1d5-2080-4493-81d5-e8616335eaf9-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:20.032511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:20.032469 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04a24cc5-qc9x7"] Oct 30 21:54:20.036936 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:20.036905 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-04a24cc5-qc9x7"] Oct 30 21:54:20.592728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:20.592694 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4" Oct 30 21:54:20.592728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:20.592707 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4" event={"ID":"89f7e1d5-2080-4493-81d5-e8616335eaf9","Type":"ContainerDied","Data":"5e54070b14933265edc78118d1b11200df73933f8a2d63e281ab33f945a23e29"} Oct 30 21:54:20.592961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:20.592767 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e54070b14933265edc78118d1b11200df73933f8a2d63e281ab33f945a23e29" Oct 30 21:54:21.017516 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:21.017472 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09de0d7b-9001-49f4-bd49-7901f130b546" path="/var/lib/kubelet/pods/09de0d7b-9001-49f4-bd49-7901f130b546/volumes" Oct 30 21:54:25.973082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:25.973028 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5"] Oct 30 21:54:25.983655 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:25.983616 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-ad73a4-d92f5"] Oct 30 21:54:27.021304 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:27.021269 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aed9789d-3a21-41f9-971b-d353d25f9ebc" path="/var/lib/kubelet/pods/aed9789d-3a21-41f9-971b-d353d25f9ebc/volumes" Oct 30 21:54:27.383050 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:27.382940 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f"] Oct 30 21:54:27.383535 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:27.383476 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" podUID="c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc" containerName="maas-api" containerID="cri-o://1ba31048818e8e42a460953e5539d7ca1f1fcc48963e7011b949614f6f2bd63b" gracePeriod=30 Oct 30 21:54:27.697946 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:27.697885 4926 generic.go:356] "Generic (PLEG): container finished" podID="c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc" containerID="1ba31048818e8e42a460953e5539d7ca1f1fcc48963e7011b949614f6f2bd63b" exitCode=0 Oct 30 21:54:27.698151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:27.698129 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" event={"ID":"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc","Type":"ContainerDied","Data":"1ba31048818e8e42a460953e5539d7ca1f1fcc48963e7011b949614f6f2bd63b"} Oct 30 21:54:27.775059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:27.773587 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" Oct 30 21:54:27.969056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:27.964924 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-metrics-certs\") pod \"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc\" (UID: \"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc\") " Oct 30 21:54:27.969056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:27.965002 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-openshift-service-ca\") pod \"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc\" (UID: \"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc\") " Oct 30 21:54:27.969056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:27.965114 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-kube-api-access-ctmqf\" (UniqueName: \"kubernetes.io/projected/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-kube-api-access-ctmqf\") pod \"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc\" (UID: \"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc\") " Oct 30 21:54:27.969056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:27.965330 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-maas-api-tls\") pod \"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc\" (UID: \"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc\") " Oct 30 21:54:28.023652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:28.023612 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-maas-api-tls" pod "c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc" (UID: "c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:28.038397 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:28.036529 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-kube-api-access-ctmqf" pod "c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc" (UID: "c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc"). InnerVolumeSpecName "kube-api-access-ctmqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:28.054604 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:28.054557 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-metrics-certs" pod "c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc" (UID: "c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:28.072752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:28.072707 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-openshift-service-ca" pod "c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc" (UID: "c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:28.074709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:28.074676 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-maas-api-tls\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:28.074709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:28.074710 4926 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-metrics-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:28.074961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:28.074725 4926 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-openshift-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:28.074961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:28.074739 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctmqf\" (UniqueName: \"kubernetes.io/projected/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc-kube-api-access-ctmqf\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:28.116891 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:28.116850 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4"] Oct 30 21:54:28.135999 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:28.124911 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e136b8-4ndq4"] Oct 30 21:54:28.710926 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:28.710846 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" Oct 30 21:54:28.711231 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:28.710871 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f" event={"ID":"c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc","Type":"ContainerDied","Data":"6fa370107bd559aef85c18ab2790c6c007bb8c47c683173e581a00b841091392"} Oct 30 21:54:28.711348 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:28.711268 4926 scope.go:122] "RemoveContainer" containerID="1ba31048818e8e42a460953e5539d7ca1f1fcc48963e7011b949614f6f2bd63b" Oct 30 21:54:28.770606 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:28.770566 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f"] Oct 30 21:54:28.781526 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:28.781494 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e136b8-657db44ff4-9fx6f"] Oct 30 21:54:29.019998 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:29.019898 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89f7e1d5-2080-4493-81d5-e8616335eaf9" path="/var/lib/kubelet/pods/89f7e1d5-2080-4493-81d5-e8616335eaf9/volumes" Oct 30 21:54:29.020559 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:29.020533 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc" path="/var/lib/kubelet/pods/c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc/volumes" Oct 30 21:54:36.113421 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.113380 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg"] Oct 30 21:54:36.114273 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.114248 4926 cpu_manager.go:414] "containerMap: removing container" podUID="89f7e1d5-2080-4493-81d5-e8616335eaf9" containerName="revoke-keys" Oct 30 21:54:36.114273 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.114271 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89f7e1d5-2080-4493-81d5-e8616335eaf9" containerName="revoke-keys" Oct 30 21:54:36.114451 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.114284 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc" containerName="maas-api" Oct 30 21:54:36.114451 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.114290 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc" containerName="maas-api" Oct 30 21:54:36.114451 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.114409 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="89f7e1d5-2080-4493-81d5-e8616335eaf9" containerName="revoke-keys" Oct 30 21:54:36.114451 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.114419 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c89ebf4c-90ed-41d6-b85d-e59d44e3ebbc" containerName="maas-api" Oct 30 21:54:36.119826 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.119803 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg" Oct 30 21:54:36.134881 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.134850 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg"] Oct 30 21:54:36.143407 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.139686 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:54:36.143407 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.139738 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:36.246106 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.246065 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h849m\" (UniqueName: \"kubernetes.io/projected/3727d30f-3bd6-40c3-a86c-9a2f1896a06f-kube-api-access-h849m\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg\" (UID: \"3727d30f-3bd6-40c3-a86c-9a2f1896a06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg" Oct 30 21:54:36.246318 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.246270 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3727d30f-3bd6-40c3-a86c-9a2f1896a06f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg\" (UID: \"3727d30f-3bd6-40c3-a86c-9a2f1896a06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg" Oct 30 21:54:36.246411 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.246394 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3727d30f-3bd6-40c3-a86c-9a2f1896a06f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg\" (UID: \"3727d30f-3bd6-40c3-a86c-9a2f1896a06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg" Oct 30 21:54:36.353316 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.353275 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h849m\" (UniqueName: \"kubernetes.io/projected/3727d30f-3bd6-40c3-a86c-9a2f1896a06f-kube-api-access-h849m\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg\" (UID: \"3727d30f-3bd6-40c3-a86c-9a2f1896a06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg" Oct 30 21:54:36.354088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.354054 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3727d30f-3bd6-40c3-a86c-9a2f1896a06f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg\" (UID: \"3727d30f-3bd6-40c3-a86c-9a2f1896a06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg" Oct 30 21:54:36.355085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.355061 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3727d30f-3bd6-40c3-a86c-9a2f1896a06f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg\" (UID: \"3727d30f-3bd6-40c3-a86c-9a2f1896a06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg" Oct 30 21:54:36.356345 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.356124 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3727d30f-3bd6-40c3-a86c-9a2f1896a06f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg\" (UID: \"3727d30f-3bd6-40c3-a86c-9a2f1896a06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg" Oct 30 21:54:36.404753 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.404674 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3727d30f-3bd6-40c3-a86c-9a2f1896a06f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg\" (UID: \"3727d30f-3bd6-40c3-a86c-9a2f1896a06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg" Oct 30 21:54:36.413748 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.413409 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h849m\" (UniqueName: \"kubernetes.io/projected/3727d30f-3bd6-40c3-a86c-9a2f1896a06f-kube-api-access-h849m\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg\" (UID: \"3727d30f-3bd6-40c3-a86c-9a2f1896a06f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg" Oct 30 21:54:36.436474 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.436440 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg" Oct 30 21:54:36.507574 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.507531 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76"] Oct 30 21:54:36.522354 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.522324 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76" Oct 30 21:54:36.543208 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.541282 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76"] Oct 30 21:54:36.661150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.661021 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b32b9802-d274-43ca-a973-1a17b0dfcc72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76\" (UID: \"b32b9802-d274-43ca-a973-1a17b0dfcc72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76" Oct 30 21:54:36.661362 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.661209 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xz6q\" (UniqueName: \"kubernetes.io/projected/b32b9802-d274-43ca-a973-1a17b0dfcc72-kube-api-access-2xz6q\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76\" (UID: \"b32b9802-d274-43ca-a973-1a17b0dfcc72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76" Oct 30 21:54:36.661362 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.661321 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b32b9802-d274-43ca-a973-1a17b0dfcc72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76\" (UID: \"b32b9802-d274-43ca-a973-1a17b0dfcc72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76" Oct 30 21:54:36.766857 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.766561 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b32b9802-d274-43ca-a973-1a17b0dfcc72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76\" (UID: \"b32b9802-d274-43ca-a973-1a17b0dfcc72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76" Oct 30 21:54:36.766857 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.766743 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b32b9802-d274-43ca-a973-1a17b0dfcc72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76\" (UID: \"b32b9802-d274-43ca-a973-1a17b0dfcc72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76" Oct 30 21:54:36.766857 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.766839 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xz6q\" (UniqueName: \"kubernetes.io/projected/b32b9802-d274-43ca-a973-1a17b0dfcc72-kube-api-access-2xz6q\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76\" (UID: \"b32b9802-d274-43ca-a973-1a17b0dfcc72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76" Oct 30 21:54:36.768328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.768301 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b32b9802-d274-43ca-a973-1a17b0dfcc72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76\" (UID: \"b32b9802-d274-43ca-a973-1a17b0dfcc72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76" Oct 30 21:54:36.770496 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:54:36.770456 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3727d30f_3bd6_40c3_a86c_9a2f1896a06f.slice/crio-46e07b83dc9e8206ac88ca3ce792a5bc5a250ada31af625ec410efa5358a6257 WatchSource:0}: Error finding container 46e07b83dc9e8206ac88ca3ce792a5bc5a250ada31af625ec410efa5358a6257: Status 404 returned error can't find the container with id 46e07b83dc9e8206ac88ca3ce792a5bc5a250ada31af625ec410efa5358a6257 Oct 30 21:54:36.780542 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.780248 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg"] Oct 30 21:54:36.782876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.782844 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b32b9802-d274-43ca-a973-1a17b0dfcc72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76\" (UID: \"b32b9802-d274-43ca-a973-1a17b0dfcc72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76" Oct 30 21:54:36.786682 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.786649 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xz6q\" (UniqueName: \"kubernetes.io/projected/b32b9802-d274-43ca-a973-1a17b0dfcc72-kube-api-access-2xz6q\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76\" (UID: \"b32b9802-d274-43ca-a973-1a17b0dfcc72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76" Oct 30 21:54:36.858586 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.858543 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg" event={"ID":"3727d30f-3bd6-40c3-a86c-9a2f1896a06f","Type":"ContainerStarted","Data":"46e07b83dc9e8206ac88ca3ce792a5bc5a250ada31af625ec410efa5358a6257"} Oct 30 21:54:36.867798 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:36.867760 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76" Oct 30 21:54:37.211307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:37.210709 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76"] Oct 30 21:54:37.216874 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:54:37.216841 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32b9802_d274_43ca_a973_1a17b0dfcc72.slice/crio-298f40d670e7b97fa8335b6deb3d220ee90e3c715ef8759e2e3fa3b6d7770bf8 WatchSource:0}: Error finding container 298f40d670e7b97fa8335b6deb3d220ee90e3c715ef8759e2e3fa3b6d7770bf8: Status 404 returned error can't find the container with id 298f40d670e7b97fa8335b6deb3d220ee90e3c715ef8759e2e3fa3b6d7770bf8 Oct 30 21:54:37.232474 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:37.231691 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c"] Oct 30 21:54:37.232474 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:37.231927 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" podUID="e89edd0d-00f1-4a05-baa6-b053af519b55" containerName="istio-proxy" containerID="cri-o://4e744ecfcc14b3caca4d66b13cf5fca50d17a959f389770b10003d575266fb6b" gracePeriod=30 Oct 30 21:54:37.501422 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:54:37.501379 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:54:37.792341 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:37.792250 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f"] Oct 30 21:54:37.797162 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:37.797124 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f" Oct 30 21:54:37.815760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:37.815727 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f"] Oct 30 21:54:37.869714 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:37.869664 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg" event={"ID":"3727d30f-3bd6-40c3-a86c-9a2f1896a06f","Type":"ContainerStarted","Data":"7fe1675698a12817ae72aa0f335e2da13b46bfa827014cfa35d1a1cb2f5d839d"} Oct 30 21:54:37.872536 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:37.872493 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76" event={"ID":"b32b9802-d274-43ca-a973-1a17b0dfcc72","Type":"ContainerStarted","Data":"1f687370f4b3a4ae555a54472fd8b0d94f4fb89282345330fe348f5f0286a9ec"} Oct 30 21:54:37.872735 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:37.872552 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76" event={"ID":"b32b9802-d274-43ca-a973-1a17b0dfcc72","Type":"ContainerStarted","Data":"298f40d670e7b97fa8335b6deb3d220ee90e3c715ef8759e2e3fa3b6d7770bf8"} Oct 30 21:54:37.899155 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:37.899118 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/398b8d42-6bcc-4a61-8807-05fbf6d9df47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f\" (UID: \"398b8d42-6bcc-4a61-8807-05fbf6d9df47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f" Oct 30 21:54:37.899349 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:37.899171 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/398b8d42-6bcc-4a61-8807-05fbf6d9df47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f\" (UID: \"398b8d42-6bcc-4a61-8807-05fbf6d9df47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f" Oct 30 21:54:37.899415 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:37.899349 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnnms\" (UniqueName: \"kubernetes.io/projected/398b8d42-6bcc-4a61-8807-05fbf6d9df47-kube-api-access-jnnms\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f\" (UID: \"398b8d42-6bcc-4a61-8807-05fbf6d9df47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f" Oct 30 21:54:37.913943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:37.913881 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg" podStartSLOduration=1.9138617660000001 podStartE2EDuration="1.913861766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:37.907718564 +0000 UTC m=+3265.028792310" watchObservedRunningTime="2026-10-30 21:54:37.913861766 +0000 UTC m=+3265.034935513" Oct 30 21:54:37.946862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:37.946810 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76" podStartSLOduration=1.946796969 podStartE2EDuration="1.946796969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:37.945284423 +0000 UTC m=+3265.066358171" watchObservedRunningTime="2026-10-30 21:54:37.946796969 +0000 UTC m=+3265.067870770" Oct 30 21:54:38.000846 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:38.000805 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/398b8d42-6bcc-4a61-8807-05fbf6d9df47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f\" (UID: \"398b8d42-6bcc-4a61-8807-05fbf6d9df47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f" Oct 30 21:54:38.001054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:38.000887 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/398b8d42-6bcc-4a61-8807-05fbf6d9df47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f\" (UID: \"398b8d42-6bcc-4a61-8807-05fbf6d9df47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f" Oct 30 21:54:38.001131 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:38.001112 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnnms\" (UniqueName: \"kubernetes.io/projected/398b8d42-6bcc-4a61-8807-05fbf6d9df47-kube-api-access-jnnms\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f\" (UID: \"398b8d42-6bcc-4a61-8807-05fbf6d9df47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f" Oct 30 21:54:38.002262 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:38.002230 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/398b8d42-6bcc-4a61-8807-05fbf6d9df47-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f\" (UID: \"398b8d42-6bcc-4a61-8807-05fbf6d9df47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f" Oct 30 21:54:38.003757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:38.003727 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/398b8d42-6bcc-4a61-8807-05fbf6d9df47-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f\" (UID: \"398b8d42-6bcc-4a61-8807-05fbf6d9df47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f" Oct 30 21:54:38.019219 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:38.019191 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnnms\" (UniqueName: \"kubernetes.io/projected/398b8d42-6bcc-4a61-8807-05fbf6d9df47-kube-api-access-jnnms\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f\" (UID: \"398b8d42-6bcc-4a61-8807-05fbf6d9df47\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f" Oct 30 21:54:38.113745 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:38.113716 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f" Oct 30 21:54:38.442644 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:38.442607 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f"] Oct 30 21:54:38.443168 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:54:38.443141 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398b8d42_6bcc_4a61_8807_05fbf6d9df47.slice/crio-e9c2c8c14c8bab5b9573875225b8124d384f47419e1d9251c1cd7c2546702539 WatchSource:0}: Error finding container e9c2c8c14c8bab5b9573875225b8124d384f47419e1d9251c1cd7c2546702539: Status 404 returned error can't find the container with id e9c2c8c14c8bab5b9573875225b8124d384f47419e1d9251c1cd7c2546702539 Oct 30 21:54:38.884014 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:38.883977 4926 generic.go:356] "Generic (PLEG): container finished" podID="3727d30f-3bd6-40c3-a86c-9a2f1896a06f" containerID="7fe1675698a12817ae72aa0f335e2da13b46bfa827014cfa35d1a1cb2f5d839d" exitCode=0 Oct 30 21:54:38.884247 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:38.884065 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg" event={"ID":"3727d30f-3bd6-40c3-a86c-9a2f1896a06f","Type":"ContainerDied","Data":"7fe1675698a12817ae72aa0f335e2da13b46bfa827014cfa35d1a1cb2f5d839d"} Oct 30 21:54:38.885944 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:38.885913 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f" event={"ID":"398b8d42-6bcc-4a61-8807-05fbf6d9df47","Type":"ContainerStarted","Data":"e9c2c8c14c8bab5b9573875225b8124d384f47419e1d9251c1cd7c2546702539"} Oct 30 21:54:38.888364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:38.888205 4926 generic.go:356] "Generic (PLEG): container finished" podID="b32b9802-d274-43ca-a973-1a17b0dfcc72" containerID="1f687370f4b3a4ae555a54472fd8b0d94f4fb89282345330fe348f5f0286a9ec" exitCode=0 Oct 30 21:54:38.888364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:38.888247 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76" event={"ID":"b32b9802-d274-43ca-a973-1a17b0dfcc72","Type":"ContainerDied","Data":"1f687370f4b3a4ae555a54472fd8b0d94f4fb89282345330fe348f5f0286a9ec"} Oct 30 21:54:39.899891 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:39.899796 4926 generic.go:356] "Generic (PLEG): container finished" podID="398b8d42-6bcc-4a61-8807-05fbf6d9df47" containerID="2a7644386c0bb92618913f3ef4783dcc283536de6c189861a7ef47c8363dc6c2" exitCode=0 Oct 30 21:54:39.900442 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:39.899890 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f" event={"ID":"398b8d42-6bcc-4a61-8807-05fbf6d9df47","Type":"ContainerDied","Data":"2a7644386c0bb92618913f3ef4783dcc283536de6c189861a7ef47c8363dc6c2"} Oct 30 21:54:40.112123 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.112089 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76" Oct 30 21:54:40.135439 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.135407 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b32b9802-d274-43ca-a973-1a17b0dfcc72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b32b9802-d274-43ca-a973-1a17b0dfcc72-maas-api-openshift-ca\") pod \"b32b9802-d274-43ca-a973-1a17b0dfcc72\" (UID: \"b32b9802-d274-43ca-a973-1a17b0dfcc72\") " Oct 30 21:54:40.135593 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.135464 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32b9802-d274-43ca-a973-1a17b0dfcc72-kube-api-access-2xz6q\" (UniqueName: \"kubernetes.io/projected/b32b9802-d274-43ca-a973-1a17b0dfcc72-kube-api-access-2xz6q\") pod \"b32b9802-d274-43ca-a973-1a17b0dfcc72\" (UID: \"b32b9802-d274-43ca-a973-1a17b0dfcc72\") " Oct 30 21:54:40.135660 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.135652 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b32b9802-d274-43ca-a973-1a17b0dfcc72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b32b9802-d274-43ca-a973-1a17b0dfcc72-maas-api-xks-ca\") pod \"b32b9802-d274-43ca-a973-1a17b0dfcc72\" (UID: \"b32b9802-d274-43ca-a973-1a17b0dfcc72\") " Oct 30 21:54:40.135889 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.135860 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b32b9802-d274-43ca-a973-1a17b0dfcc72-maas-api-openshift-ca" pod "b32b9802-d274-43ca-a973-1a17b0dfcc72" (UID: "b32b9802-d274-43ca-a973-1a17b0dfcc72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:40.136233 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.136210 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b32b9802-d274-43ca-a973-1a17b0dfcc72-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:40.137820 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.137796 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32b9802-d274-43ca-a973-1a17b0dfcc72-kube-api-access-2xz6q" pod "b32b9802-d274-43ca-a973-1a17b0dfcc72" (UID: "b32b9802-d274-43ca-a973-1a17b0dfcc72"). InnerVolumeSpecName "kube-api-access-2xz6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:40.137943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.137892 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b32b9802-d274-43ca-a973-1a17b0dfcc72-maas-api-xks-ca" pod "b32b9802-d274-43ca-a973-1a17b0dfcc72" (UID: "b32b9802-d274-43ca-a973-1a17b0dfcc72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:40.157354 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.157284 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg" Oct 30 21:54:40.236805 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.236763 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3727d30f-3bd6-40c3-a86c-9a2f1896a06f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3727d30f-3bd6-40c3-a86c-9a2f1896a06f-maas-api-openshift-ca\") pod \"3727d30f-3bd6-40c3-a86c-9a2f1896a06f\" (UID: \"3727d30f-3bd6-40c3-a86c-9a2f1896a06f\") " Oct 30 21:54:40.236994 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.236820 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3727d30f-3bd6-40c3-a86c-9a2f1896a06f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3727d30f-3bd6-40c3-a86c-9a2f1896a06f-maas-api-xks-ca\") pod \"3727d30f-3bd6-40c3-a86c-9a2f1896a06f\" (UID: \"3727d30f-3bd6-40c3-a86c-9a2f1896a06f\") " Oct 30 21:54:40.236994 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.236905 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3727d30f-3bd6-40c3-a86c-9a2f1896a06f-kube-api-access-h849m\" (UniqueName: \"kubernetes.io/projected/3727d30f-3bd6-40c3-a86c-9a2f1896a06f-kube-api-access-h849m\") pod \"3727d30f-3bd6-40c3-a86c-9a2f1896a06f\" (UID: \"3727d30f-3bd6-40c3-a86c-9a2f1896a06f\") " Oct 30 21:54:40.237238 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.237207 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3727d30f-3bd6-40c3-a86c-9a2f1896a06f-maas-api-openshift-ca" pod "3727d30f-3bd6-40c3-a86c-9a2f1896a06f" (UID: "3727d30f-3bd6-40c3-a86c-9a2f1896a06f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:40.237720 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.237680 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3727d30f-3bd6-40c3-a86c-9a2f1896a06f-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:40.237720 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.237714 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b32b9802-d274-43ca-a973-1a17b0dfcc72-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:40.237939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.237729 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xz6q\" (UniqueName: \"kubernetes.io/projected/b32b9802-d274-43ca-a973-1a17b0dfcc72-kube-api-access-2xz6q\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:40.239276 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.239241 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3727d30f-3bd6-40c3-a86c-9a2f1896a06f-kube-api-access-h849m" pod "3727d30f-3bd6-40c3-a86c-9a2f1896a06f" (UID: "3727d30f-3bd6-40c3-a86c-9a2f1896a06f"). InnerVolumeSpecName "kube-api-access-h849m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:40.239276 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.239242 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3727d30f-3bd6-40c3-a86c-9a2f1896a06f-maas-api-xks-ca" pod "3727d30f-3bd6-40c3-a86c-9a2f1896a06f" (UID: "3727d30f-3bd6-40c3-a86c-9a2f1896a06f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:40.338799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.338754 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3727d30f-3bd6-40c3-a86c-9a2f1896a06f-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:40.338799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.338790 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h849m\" (UniqueName: \"kubernetes.io/projected/3727d30f-3bd6-40c3-a86c-9a2f1896a06f-kube-api-access-h849m\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:40.918117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.918086 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg" Oct 30 21:54:40.918607 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.918107 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg" event={"ID":"3727d30f-3bd6-40c3-a86c-9a2f1896a06f","Type":"ContainerDied","Data":"46e07b83dc9e8206ac88ca3ce792a5bc5a250ada31af625ec410efa5358a6257"} Oct 30 21:54:40.918607 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.918200 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46e07b83dc9e8206ac88ca3ce792a5bc5a250ada31af625ec410efa5358a6257" Oct 30 21:54:40.919915 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.919886 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76" Oct 30 21:54:40.920086 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.919944 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76" event={"ID":"b32b9802-d274-43ca-a973-1a17b0dfcc72","Type":"ContainerDied","Data":"298f40d670e7b97fa8335b6deb3d220ee90e3c715ef8759e2e3fa3b6d7770bf8"} Oct 30 21:54:40.920086 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:40.919993 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="298f40d670e7b97fa8335b6deb3d220ee90e3c715ef8759e2e3fa3b6d7770bf8" Oct 30 21:54:41.106764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:41.106731 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f" Oct 30 21:54:41.148682 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:41.148641 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/398b8d42-6bcc-4a61-8807-05fbf6d9df47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/398b8d42-6bcc-4a61-8807-05fbf6d9df47-maas-api-openshift-ca\") pod \"398b8d42-6bcc-4a61-8807-05fbf6d9df47\" (UID: \"398b8d42-6bcc-4a61-8807-05fbf6d9df47\") " Oct 30 21:54:41.148872 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:41.148746 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/398b8d42-6bcc-4a61-8807-05fbf6d9df47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/398b8d42-6bcc-4a61-8807-05fbf6d9df47-maas-api-xks-ca\") pod \"398b8d42-6bcc-4a61-8807-05fbf6d9df47\" (UID: \"398b8d42-6bcc-4a61-8807-05fbf6d9df47\") " Oct 30 21:54:41.148872 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:41.148828 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/398b8d42-6bcc-4a61-8807-05fbf6d9df47-kube-api-access-jnnms\" (UniqueName: \"kubernetes.io/projected/398b8d42-6bcc-4a61-8807-05fbf6d9df47-kube-api-access-jnnms\") pod \"398b8d42-6bcc-4a61-8807-05fbf6d9df47\" (UID: \"398b8d42-6bcc-4a61-8807-05fbf6d9df47\") " Oct 30 21:54:41.149011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:41.148990 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398b8d42-6bcc-4a61-8807-05fbf6d9df47-maas-api-openshift-ca" pod "398b8d42-6bcc-4a61-8807-05fbf6d9df47" (UID: "398b8d42-6bcc-4a61-8807-05fbf6d9df47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:41.149476 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:41.149447 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/398b8d42-6bcc-4a61-8807-05fbf6d9df47-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:41.151488 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:41.151461 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/398b8d42-6bcc-4a61-8807-05fbf6d9df47-maas-api-xks-ca" pod "398b8d42-6bcc-4a61-8807-05fbf6d9df47" (UID: "398b8d42-6bcc-4a61-8807-05fbf6d9df47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:41.151599 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:41.151518 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398b8d42-6bcc-4a61-8807-05fbf6d9df47-kube-api-access-jnnms" pod "398b8d42-6bcc-4a61-8807-05fbf6d9df47" (UID: "398b8d42-6bcc-4a61-8807-05fbf6d9df47"). InnerVolumeSpecName "kube-api-access-jnnms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:41.250996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:41.250913 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/398b8d42-6bcc-4a61-8807-05fbf6d9df47-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:41.250996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:41.250947 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnnms\" (UniqueName: \"kubernetes.io/projected/398b8d42-6bcc-4a61-8807-05fbf6d9df47-kube-api-access-jnnms\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:41.930605 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:41.930573 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f" Oct 30 21:54:41.931131 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:41.930601 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f" event={"ID":"398b8d42-6bcc-4a61-8807-05fbf6d9df47","Type":"ContainerDied","Data":"e9c2c8c14c8bab5b9573875225b8124d384f47419e1d9251c1cd7c2546702539"} Oct 30 21:54:41.931131 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:41.930663 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9c2c8c14c8bab5b9573875225b8124d384f47419e1d9251c1cd7c2546702539" Oct 30 21:54:42.535942 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.535913 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:54:42.569019 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.568985 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-workload-socket\") pod \"e89edd0d-00f1-4a05-baa6-b053af519b55\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " Oct 30 21:54:42.569019 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.569045 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-podinfo\") pod \"e89edd0d-00f1-4a05-baa6-b053af519b55\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " Oct 30 21:54:42.569331 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.569179 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e89edd0d-00f1-4a05-baa6-b053af519b55-kube-api-access-gtbwj\" (UniqueName: \"kubernetes.io/projected/e89edd0d-00f1-4a05-baa6-b053af519b55-kube-api-access-gtbwj\") pod \"e89edd0d-00f1-4a05-baa6-b053af519b55\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " Oct 30 21:54:42.569331 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.569223 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-workload-socket" pod "e89edd0d-00f1-4a05-baa6-b053af519b55" (UID: "e89edd0d-00f1-4a05-baa6-b053af519b55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:42.569331 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.569264 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-data\") pod \"e89edd0d-00f1-4a05-baa6-b053af519b55\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " Oct 30 21:54:42.569331 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.569323 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-envoy\") pod \"e89edd0d-00f1-4a05-baa6-b053af519b55\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " Oct 30 21:54:42.569555 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.569360 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-token\" (UniqueName: \"kubernetes.io/projected/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-token\") pod \"e89edd0d-00f1-4a05-baa6-b053af519b55\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " Oct 30 21:54:42.569555 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.569396 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-credential-socket\") pod \"e89edd0d-00f1-4a05-baa6-b053af519b55\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " Oct 30 21:54:42.569555 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.569435 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-workload-certs\") pod \"e89edd0d-00f1-4a05-baa6-b053af519b55\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " Oct 30 21:54:42.569555 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.569475 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e89edd0d-00f1-4a05-baa6-b053af519b55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e89edd0d-00f1-4a05-baa6-b053af519b55-istiod-ca-cert\") pod \"e89edd0d-00f1-4a05-baa6-b053af519b55\" (UID: \"e89edd0d-00f1-4a05-baa6-b053af519b55\") " Oct 30 21:54:42.569796 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.569652 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-credential-socket" pod "e89edd0d-00f1-4a05-baa6-b053af519b55" (UID: "e89edd0d-00f1-4a05-baa6-b053af519b55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:42.569796 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.569709 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-workload-certs" pod "e89edd0d-00f1-4a05-baa6-b053af519b55" (UID: "e89edd0d-00f1-4a05-baa6-b053af519b55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:42.569910 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.569841 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-data" pod "e89edd0d-00f1-4a05-baa6-b053af519b55" (UID: "e89edd0d-00f1-4a05-baa6-b053af519b55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:42.570148 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.570123 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89edd0d-00f1-4a05-baa6-b053af519b55-istiod-ca-cert" pod "e89edd0d-00f1-4a05-baa6-b053af519b55" (UID: "e89edd0d-00f1-4a05-baa6-b053af519b55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:42.570336 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.570285 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:42.570449 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.570342 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:42.570449 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.570355 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:42.570449 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.570368 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:42.570449 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.570381 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e89edd0d-00f1-4a05-baa6-b053af519b55-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:42.571973 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.571900 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-envoy" pod "e89edd0d-00f1-4a05-baa6-b053af519b55" (UID: "e89edd0d-00f1-4a05-baa6-b053af519b55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:54:42.572128 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.572075 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-token" pod "e89edd0d-00f1-4a05-baa6-b053af519b55" (UID: "e89edd0d-00f1-4a05-baa6-b053af519b55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:42.572128 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.572113 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89edd0d-00f1-4a05-baa6-b053af519b55-kube-api-access-gtbwj" pod "e89edd0d-00f1-4a05-baa6-b053af519b55" (UID: "e89edd0d-00f1-4a05-baa6-b053af519b55"). InnerVolumeSpecName "kube-api-access-gtbwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:42.572388 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.572368 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-podinfo" pod "e89edd0d-00f1-4a05-baa6-b053af519b55" (UID: "e89edd0d-00f1-4a05-baa6-b053af519b55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:54:42.671313 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.671270 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:42.671313 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.671308 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:42.671313 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.671321 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e89edd0d-00f1-4a05-baa6-b053af519b55-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:42.671585 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.671334 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtbwj\" (UniqueName: \"kubernetes.io/projected/e89edd0d-00f1-4a05-baa6-b053af519b55-kube-api-access-gtbwj\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:42.940862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.940823 4926 generic.go:356] "Generic (PLEG): container finished" podID="e89edd0d-00f1-4a05-baa6-b053af519b55" containerID="4e744ecfcc14b3caca4d66b13cf5fca50d17a959f389770b10003d575266fb6b" exitCode=0 Oct 30 21:54:42.941402 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.940875 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" Oct 30 21:54:42.941402 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.940910 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" event={"ID":"e89edd0d-00f1-4a05-baa6-b053af519b55","Type":"ContainerDied","Data":"4e744ecfcc14b3caca4d66b13cf5fca50d17a959f389770b10003d575266fb6b"} Oct 30 21:54:42.941402 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.940977 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c" event={"ID":"e89edd0d-00f1-4a05-baa6-b053af519b55","Type":"ContainerDied","Data":"e2c1c0ddc995b2584bc53b7c8b6e92852580a8cf1332a2eb1d42cb4ed7b1c7b9"} Oct 30 21:54:42.941402 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.941004 4926 scope.go:122] "RemoveContainer" containerID="4e744ecfcc14b3caca4d66b13cf5fca50d17a959f389770b10003d575266fb6b" Oct 30 21:54:42.954378 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.954358 4926 scope.go:122] "RemoveContainer" containerID="97dbc86ea10608782ca964f1a69998158ded69fbc3ba0cd373748be6457c0284" Oct 30 21:54:42.970505 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.970476 4926 scope.go:122] "RemoveContainer" containerID="4e744ecfcc14b3caca4d66b13cf5fca50d17a959f389770b10003d575266fb6b" Oct 30 21:54:42.970863 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:54:42.970837 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e744ecfcc14b3caca4d66b13cf5fca50d17a959f389770b10003d575266fb6b\": container with ID starting with 4e744ecfcc14b3caca4d66b13cf5fca50d17a959f389770b10003d575266fb6b not found: ID does not exist" containerID="4e744ecfcc14b3caca4d66b13cf5fca50d17a959f389770b10003d575266fb6b" Oct 30 21:54:42.970956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.970876 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e744ecfcc14b3caca4d66b13cf5fca50d17a959f389770b10003d575266fb6b"} err="failed to get container status \"4e744ecfcc14b3caca4d66b13cf5fca50d17a959f389770b10003d575266fb6b\": rpc error: code = NotFound desc = could not find container \"4e744ecfcc14b3caca4d66b13cf5fca50d17a959f389770b10003d575266fb6b\": container with ID starting with 4e744ecfcc14b3caca4d66b13cf5fca50d17a959f389770b10003d575266fb6b not found: ID does not exist" Oct 30 21:54:42.970956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.970901 4926 scope.go:122] "RemoveContainer" containerID="97dbc86ea10608782ca964f1a69998158ded69fbc3ba0cd373748be6457c0284" Oct 30 21:54:42.971269 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:54:42.971244 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97dbc86ea10608782ca964f1a69998158ded69fbc3ba0cd373748be6457c0284\": container with ID starting with 97dbc86ea10608782ca964f1a69998158ded69fbc3ba0cd373748be6457c0284 not found: ID does not exist" containerID="97dbc86ea10608782ca964f1a69998158ded69fbc3ba0cd373748be6457c0284" Oct 30 21:54:42.971328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:42.971279 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97dbc86ea10608782ca964f1a69998158ded69fbc3ba0cd373748be6457c0284"} err="failed to get container status \"97dbc86ea10608782ca964f1a69998158ded69fbc3ba0cd373748be6457c0284\": rpc error: code = NotFound desc = could not find container \"97dbc86ea10608782ca964f1a69998158ded69fbc3ba0cd373748be6457c0284\": container with ID starting with 97dbc86ea10608782ca964f1a69998158ded69fbc3ba0cd373748be6457c0284 not found: ID does not exist" Oct 30 21:54:43.003613 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:43.002917 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c"] Oct 30 21:54:43.008351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:43.008311 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e136b8-openshift-default-84c5d67df9-5fs2c"] Oct 30 21:54:43.030065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:43.030014 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e89edd0d-00f1-4a05-baa6-b053af519b55" path="/var/lib/kubelet/pods/e89edd0d-00f1-4a05-baa6-b053af519b55/volumes" Oct 30 21:54:47.249363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:47.249317 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv"] Oct 30 21:54:48.093682 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:48.093641 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg"] Oct 30 21:54:48.097764 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:48.097733 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-70d3a9-np5lg"] Oct 30 21:54:49.021400 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:49.021363 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3727d30f-3bd6-40c3-a86c-9a2f1896a06f" path="/var/lib/kubelet/pods/3727d30f-3bd6-40c3-a86c-9a2f1896a06f/volumes" Oct 30 21:54:50.364187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.364151 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b"] Oct 30 21:54:50.364742 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.364727 4926 cpu_manager.go:414] "containerMap: removing container" podUID="e89edd0d-00f1-4a05-baa6-b053af519b55" containerName="istio-proxy" Oct 30 21:54:50.364790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.364742 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89edd0d-00f1-4a05-baa6-b053af519b55" containerName="istio-proxy" Oct 30 21:54:50.364790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.364776 4926 cpu_manager.go:414] "containerMap: removing container" podUID="3727d30f-3bd6-40c3-a86c-9a2f1896a06f" containerName="revoke-keys" Oct 30 21:54:50.364790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.364781 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3727d30f-3bd6-40c3-a86c-9a2f1896a06f" containerName="revoke-keys" Oct 30 21:54:50.364892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.364795 4926 cpu_manager.go:414] "containerMap: removing container" podUID="b32b9802-d274-43ca-a973-1a17b0dfcc72" containerName="revoke-keys" Oct 30 21:54:50.364892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.364801 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b32b9802-d274-43ca-a973-1a17b0dfcc72" containerName="revoke-keys" Oct 30 21:54:50.364892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.364837 4926 cpu_manager.go:414] "containerMap: removing container" podUID="398b8d42-6bcc-4a61-8807-05fbf6d9df47" containerName="revoke-keys" Oct 30 21:54:50.364892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.364843 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="398b8d42-6bcc-4a61-8807-05fbf6d9df47" containerName="revoke-keys" Oct 30 21:54:50.365023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.364931 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="3727d30f-3bd6-40c3-a86c-9a2f1896a06f" containerName="revoke-keys" Oct 30 21:54:50.365023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.364943 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89edd0d-00f1-4a05-baa6-b053af519b55" containerName="istio-proxy" Oct 30 21:54:50.365023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.364949 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="b32b9802-d274-43ca-a973-1a17b0dfcc72" containerName="revoke-keys" Oct 30 21:54:50.365023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.364959 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="398b8d42-6bcc-4a61-8807-05fbf6d9df47" containerName="revoke-keys" Oct 30 21:54:50.365023 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.364965 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89edd0d-00f1-4a05-baa6-b053af519b55" containerName="istio-proxy" Oct 30 21:54:50.368920 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.368899 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b" Oct 30 21:54:50.388085 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.388055 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b"] Oct 30 21:54:50.394090 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.392274 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:50.394090 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.392358 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:54:50.406435 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.406403 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz7p7\" (UniqueName: \"kubernetes.io/projected/67c5b659-8710-4d3f-8bc9-66b6579b26a7-kube-api-access-tz7p7\") pod \"maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b\" (UID: \"67c5b659-8710-4d3f-8bc9-66b6579b26a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b" Oct 30 21:54:50.406634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.406614 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c5b659-8710-4d3f-8bc9-66b6579b26a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b\" (UID: \"67c5b659-8710-4d3f-8bc9-66b6579b26a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b" Oct 30 21:54:50.406822 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.406796 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c5b659-8710-4d3f-8bc9-66b6579b26a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b\" (UID: \"67c5b659-8710-4d3f-8bc9-66b6579b26a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b" Oct 30 21:54:50.508528 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.508475 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c5b659-8710-4d3f-8bc9-66b6579b26a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b\" (UID: \"67c5b659-8710-4d3f-8bc9-66b6579b26a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b" Oct 30 21:54:50.508760 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.508638 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz7p7\" (UniqueName: \"kubernetes.io/projected/67c5b659-8710-4d3f-8bc9-66b6579b26a7-kube-api-access-tz7p7\") pod \"maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b\" (UID: \"67c5b659-8710-4d3f-8bc9-66b6579b26a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b" Oct 30 21:54:50.508825 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.508809 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c5b659-8710-4d3f-8bc9-66b6579b26a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b\" (UID: \"67c5b659-8710-4d3f-8bc9-66b6579b26a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b" Oct 30 21:54:50.509532 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.509503 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c5b659-8710-4d3f-8bc9-66b6579b26a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b\" (UID: \"67c5b659-8710-4d3f-8bc9-66b6579b26a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b" Oct 30 21:54:50.511194 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.511171 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c5b659-8710-4d3f-8bc9-66b6579b26a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b\" (UID: \"67c5b659-8710-4d3f-8bc9-66b6579b26a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b" Oct 30 21:54:50.533149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.533117 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz7p7\" (UniqueName: \"kubernetes.io/projected/67c5b659-8710-4d3f-8bc9-66b6579b26a7-kube-api-access-tz7p7\") pod \"maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b\" (UID: \"67c5b659-8710-4d3f-8bc9-66b6579b26a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b" Oct 30 21:54:50.688536 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:50.688498 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b" Oct 30 21:54:51.022499 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:51.022470 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b"] Oct 30 21:54:51.023087 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:54:51.023026 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c5b659_8710_4d3f_8bc9_66b6579b26a7.slice/crio-501a6ab3305342eb5ca00794451d1ad45998ad900522dfa28318e82dfb942f43 WatchSource:0}: Error finding container 501a6ab3305342eb5ca00794451d1ad45998ad900522dfa28318e82dfb942f43: Status 404 returned error can't find the container with id 501a6ab3305342eb5ca00794451d1ad45998ad900522dfa28318e82dfb942f43 Oct 30 21:54:51.046534 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:51.046500 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b" event={"ID":"67c5b659-8710-4d3f-8bc9-66b6579b26a7","Type":"ContainerStarted","Data":"501a6ab3305342eb5ca00794451d1ad45998ad900522dfa28318e82dfb942f43"} Oct 30 21:54:52.056451 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:52.056410 4926 generic.go:356] "Generic (PLEG): container finished" podID="67c5b659-8710-4d3f-8bc9-66b6579b26a7" containerID="55b80b19f16d3b086b9f5323cce7d8055fa518cac4bd72aa13edd2c40dd302ba" exitCode=0 Oct 30 21:54:52.056850 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:52.056498 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b" event={"ID":"67c5b659-8710-4d3f-8bc9-66b6579b26a7","Type":"ContainerDied","Data":"55b80b19f16d3b086b9f5323cce7d8055fa518cac4bd72aa13edd2c40dd302ba"} Oct 30 21:54:52.250726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:52.250628 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv" podUID="1210d99e-e106-4c6e-8947-cb54d3ec4c2e" containerName="payload-processing" containerID="cri-o://22714bbf960eae8afff4887f37d9d26b2306f61e8f15f438c652665d2c7f8b97" gracePeriod=25 Oct 30 21:54:53.252168 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:53.252131 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b" Oct 30 21:54:53.355722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:53.355631 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67c5b659-8710-4d3f-8bc9-66b6579b26a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c5b659-8710-4d3f-8bc9-66b6579b26a7-maas-api-xks-ca\") pod \"67c5b659-8710-4d3f-8bc9-66b6579b26a7\" (UID: \"67c5b659-8710-4d3f-8bc9-66b6579b26a7\") " Oct 30 21:54:53.355722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:53.355693 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67c5b659-8710-4d3f-8bc9-66b6579b26a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c5b659-8710-4d3f-8bc9-66b6579b26a7-maas-api-openshift-ca\") pod \"67c5b659-8710-4d3f-8bc9-66b6579b26a7\" (UID: \"67c5b659-8710-4d3f-8bc9-66b6579b26a7\") " Oct 30 21:54:53.355922 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:53.355811 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c5b659-8710-4d3f-8bc9-66b6579b26a7-kube-api-access-tz7p7\" (UniqueName: \"kubernetes.io/projected/67c5b659-8710-4d3f-8bc9-66b6579b26a7-kube-api-access-tz7p7\") pod \"67c5b659-8710-4d3f-8bc9-66b6579b26a7\" (UID: \"67c5b659-8710-4d3f-8bc9-66b6579b26a7\") " Oct 30 21:54:53.356113 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:53.356086 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67c5b659-8710-4d3f-8bc9-66b6579b26a7-maas-api-openshift-ca" pod "67c5b659-8710-4d3f-8bc9-66b6579b26a7" (UID: "67c5b659-8710-4d3f-8bc9-66b6579b26a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:54:53.356474 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:53.356448 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c5b659-8710-4d3f-8bc9-66b6579b26a7-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:53.358062 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:53.358002 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c5b659-8710-4d3f-8bc9-66b6579b26a7-kube-api-access-tz7p7" pod "67c5b659-8710-4d3f-8bc9-66b6579b26a7" (UID: "67c5b659-8710-4d3f-8bc9-66b6579b26a7"). InnerVolumeSpecName "kube-api-access-tz7p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:54:53.358153 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:53.358066 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67c5b659-8710-4d3f-8bc9-66b6579b26a7-maas-api-xks-ca" pod "67c5b659-8710-4d3f-8bc9-66b6579b26a7" (UID: "67c5b659-8710-4d3f-8bc9-66b6579b26a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:54:53.458223 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:53.458187 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz7p7\" (UniqueName: \"kubernetes.io/projected/67c5b659-8710-4d3f-8bc9-66b6579b26a7-kube-api-access-tz7p7\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:53.458223 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:53.458219 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c5b659-8710-4d3f-8bc9-66b6579b26a7-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:54:54.074611 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:54.074575 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b" Oct 30 21:54:54.074804 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:54.074621 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b" event={"ID":"67c5b659-8710-4d3f-8bc9-66b6579b26a7","Type":"ContainerDied","Data":"501a6ab3305342eb5ca00794451d1ad45998ad900522dfa28318e82dfb942f43"} Oct 30 21:54:54.074804 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:54.074681 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="501a6ab3305342eb5ca00794451d1ad45998ad900522dfa28318e82dfb942f43" Oct 30 21:54:56.044256 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:56.044164 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv" podUID="1210d99e-e106-4c6e-8947-cb54d3ec4c2e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:54:56.159898 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:56.159855 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp"] Oct 30 21:54:56.160637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:56.160604 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" podUID="d3d50005-aee2-4d9f-b966-7f8a41df5a5a" containerName="istio-proxy" containerID="cri-o://3c6f0544f64cd19b22520827eb6bb494235131d7ad88350d3a6cf3bb4c09312f" gracePeriod=30 Oct 30 21:54:56.870794 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:56.870755 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk"] Oct 30 21:54:56.871349 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:56.871334 4926 cpu_manager.go:414] "containerMap: removing container" podUID="e89edd0d-00f1-4a05-baa6-b053af519b55" containerName="istio-proxy" Oct 30 21:54:56.871469 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:56.871350 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89edd0d-00f1-4a05-baa6-b053af519b55" containerName="istio-proxy" Oct 30 21:54:56.871469 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:56.871430 4926 cpu_manager.go:414] "containerMap: removing container" podUID="67c5b659-8710-4d3f-8bc9-66b6579b26a7" containerName="revoke-keys" Oct 30 21:54:56.871469 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:56.871437 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67c5b659-8710-4d3f-8bc9-66b6579b26a7" containerName="revoke-keys" Oct 30 21:54:56.871655 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:56.871549 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="67c5b659-8710-4d3f-8bc9-66b6579b26a7" containerName="revoke-keys" Oct 30 21:54:56.875759 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:56.875735 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk" Oct 30 21:54:56.899490 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:56.899452 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:54:56.899673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:56.899528 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:54:56.907011 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:56.906976 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk"] Oct 30 21:54:57.042533 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.042493 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzshj\" (UniqueName: \"kubernetes.io/projected/c344af02-b407-421d-aea3-6f3789ee9017-kube-api-access-jzshj\") pod \"maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk\" (UID: \"c344af02-b407-421d-aea3-6f3789ee9017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk" Oct 30 21:54:57.042806 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.042656 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c344af02-b407-421d-aea3-6f3789ee9017-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk\" (UID: \"c344af02-b407-421d-aea3-6f3789ee9017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk" Oct 30 21:54:57.042806 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.042688 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c344af02-b407-421d-aea3-6f3789ee9017-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk\" (UID: \"c344af02-b407-421d-aea3-6f3789ee9017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk" Oct 30 21:54:57.144557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.144466 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c344af02-b407-421d-aea3-6f3789ee9017-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk\" (UID: \"c344af02-b407-421d-aea3-6f3789ee9017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk" Oct 30 21:54:57.144557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.144510 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c344af02-b407-421d-aea3-6f3789ee9017-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk\" (UID: \"c344af02-b407-421d-aea3-6f3789ee9017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk" Oct 30 21:54:57.145150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.144678 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzshj\" (UniqueName: \"kubernetes.io/projected/c344af02-b407-421d-aea3-6f3789ee9017-kube-api-access-jzshj\") pod \"maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk\" (UID: \"c344af02-b407-421d-aea3-6f3789ee9017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk" Oct 30 21:54:57.145451 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.145414 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c344af02-b407-421d-aea3-6f3789ee9017-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk\" (UID: \"c344af02-b407-421d-aea3-6f3789ee9017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk" Oct 30 21:54:57.147278 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.147245 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c344af02-b407-421d-aea3-6f3789ee9017-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk\" (UID: \"c344af02-b407-421d-aea3-6f3789ee9017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk" Oct 30 21:54:57.166612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.166585 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzshj\" (UniqueName: \"kubernetes.io/projected/c344af02-b407-421d-aea3-6f3789ee9017-kube-api-access-jzshj\") pod \"maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk\" (UID: \"c344af02-b407-421d-aea3-6f3789ee9017\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk" Oct 30 21:54:57.193292 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.193263 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk" Oct 30 21:54:57.417636 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.417598 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj"] Oct 30 21:54:57.423232 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.423209 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.445574 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.445546 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-889195-openshift-default-dockercfg-l5cnq\"" Oct 30 21:54:57.462964 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.462932 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj"] Oct 30 21:54:57.515326 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.515287 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk"] Oct 30 21:54:57.550396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.550365 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-workload-socket\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.550534 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.550409 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-envoy\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.550534 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.550433 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-token\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.550612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.550547 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-credential-socket\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.550612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.550600 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvpz9\" (UniqueName: \"kubernetes.io/projected/3bf71b99-a679-4d29-b081-5d26c7abfcde-kube-api-access-tvpz9\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.550686 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.550677 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-podinfo\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.550730 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.550717 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-data\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.550844 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.550825 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf71b99-a679-4d29-b081-5d26c7abfcde-istiod-ca-cert\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.550886 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.550875 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-workload-certs\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.652095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.652007 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-workload-socket\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.652287 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.652121 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-envoy\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.652287 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.652157 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-token\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.652287 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.652186 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-credential-socket\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.652287 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.652244 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvpz9\" (UniqueName: \"kubernetes.io/projected/3bf71b99-a679-4d29-b081-5d26c7abfcde-kube-api-access-tvpz9\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.652287 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.652273 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-podinfo\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.652566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.652337 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-data\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.652566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.652428 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf71b99-a679-4d29-b081-5d26c7abfcde-istiod-ca-cert\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.652566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.652489 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-workload-certs\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.652566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.652520 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-workload-socket\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.652807 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.652724 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-credential-socket\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.652866 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.652810 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-data\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.652910 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.652881 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-workload-certs\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.653266 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.653248 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf71b99-a679-4d29-b081-5d26c7abfcde-istiod-ca-cert\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.654688 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.654664 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-envoy\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.654892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.654871 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-podinfo\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.675541 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.675469 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-token\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.677120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.677093 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvpz9\" (UniqueName: \"kubernetes.io/projected/3bf71b99-a679-4d29-b081-5d26c7abfcde-kube-api-access-tvpz9\") pod \"e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:57.739083 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:57.739045 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:58.076874 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:58.076819 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj"] Oct 30 21:54:58.080583 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:54:58.080547 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf71b99_a679_4d29_b081_5d26c7abfcde.slice/crio-555e28684bb3402e83db47849552d7358ca6d7b7f4fa5cda1ed63421563462f6 WatchSource:0}: Error finding container 555e28684bb3402e83db47849552d7358ca6d7b7f4fa5cda1ed63421563462f6: Status 404 returned error can't find the container with id 555e28684bb3402e83db47849552d7358ca6d7b7f4fa5cda1ed63421563462f6 Oct 30 21:54:58.084180 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:58.084118 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:54:58.084292 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:58.084232 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:54:58.084292 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:58.084279 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 21:54:58.161745 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:58.161707 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk" event={"ID":"c344af02-b407-421d-aea3-6f3789ee9017","Type":"ContainerStarted","Data":"3ea7750fada069e75036b6e7fd34d480ba23a2222a15282cb36b732227ab742c"} Oct 30 21:54:58.162191 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:58.161777 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk" event={"ID":"c344af02-b407-421d-aea3-6f3789ee9017","Type":"ContainerStarted","Data":"b04f41d039916c960fa680251b3849beff51f8b70df175ec1ba13240ec2b9f82"} Oct 30 21:54:58.163687 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:58.163661 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" event={"ID":"3bf71b99-a679-4d29-b081-5d26c7abfcde","Type":"ContainerStarted","Data":"555e28684bb3402e83db47849552d7358ca6d7b7f4fa5cda1ed63421563462f6"} Oct 30 21:54:58.235983 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:58.235859 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk" podStartSLOduration=2.235836688 podStartE2EDuration="2.235836688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:58.222783173 +0000 UTC m=+3285.343856920" watchObservedRunningTime="2026-10-30 21:54:58.235836688 +0000 UTC m=+3285.356910435" Oct 30 21:54:58.311302 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:58.311235 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" podStartSLOduration=1.3112163319999999 podStartE2EDuration="1.311216332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:58.30846335 +0000 UTC m=+3285.429537099" watchObservedRunningTime="2026-10-30 21:54:58.311216332 +0000 UTC m=+3285.432290080" Oct 30 21:54:58.739692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:58.739652 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:54:59.210955 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:59.210916 4926 generic.go:356] "Generic (PLEG): container finished" podID="c344af02-b407-421d-aea3-6f3789ee9017" containerID="3ea7750fada069e75036b6e7fd34d480ba23a2222a15282cb36b732227ab742c" exitCode=0 Oct 30 21:54:59.211708 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:59.211130 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk" event={"ID":"c344af02-b407-421d-aea3-6f3789ee9017","Type":"ContainerDied","Data":"3ea7750fada069e75036b6e7fd34d480ba23a2222a15282cb36b732227ab742c"} Oct 30 21:54:59.214273 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:59.214239 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" event={"ID":"3bf71b99-a679-4d29-b081-5d26c7abfcde","Type":"ContainerStarted","Data":"0580a012b0114b0f9f738092a9f4996c1f47990f18404e33143fb6d81782a20e"} Oct 30 21:54:59.741523 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:59.741481 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Oct 30 21:54:59.741746 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:54:59.741550 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" podUID="3bf71b99-a679-4d29-b081-5d26c7abfcde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Oct 30 21:55:00.505361 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:00.505280 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk" Oct 30 21:55:00.525245 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:00.525207 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c344af02-b407-421d-aea3-6f3789ee9017-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c344af02-b407-421d-aea3-6f3789ee9017-maas-api-xks-ca\") pod \"c344af02-b407-421d-aea3-6f3789ee9017\" (UID: \"c344af02-b407-421d-aea3-6f3789ee9017\") " Oct 30 21:55:00.525455 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:00.525294 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c344af02-b407-421d-aea3-6f3789ee9017-kube-api-access-jzshj\" (UniqueName: \"kubernetes.io/projected/c344af02-b407-421d-aea3-6f3789ee9017-kube-api-access-jzshj\") pod \"c344af02-b407-421d-aea3-6f3789ee9017\" (UID: \"c344af02-b407-421d-aea3-6f3789ee9017\") " Oct 30 21:55:00.525455 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:00.525372 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c344af02-b407-421d-aea3-6f3789ee9017-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c344af02-b407-421d-aea3-6f3789ee9017-maas-api-openshift-ca\") pod \"c344af02-b407-421d-aea3-6f3789ee9017\" (UID: \"c344af02-b407-421d-aea3-6f3789ee9017\") " Oct 30 21:55:00.526169 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:00.526121 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c344af02-b407-421d-aea3-6f3789ee9017-maas-api-openshift-ca" pod "c344af02-b407-421d-aea3-6f3789ee9017" (UID: "c344af02-b407-421d-aea3-6f3789ee9017"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:00.527475 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:00.527441 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c344af02-b407-421d-aea3-6f3789ee9017-maas-api-xks-ca" pod "c344af02-b407-421d-aea3-6f3789ee9017" (UID: "c344af02-b407-421d-aea3-6f3789ee9017"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:00.528120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:00.528086 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c344af02-b407-421d-aea3-6f3789ee9017-kube-api-access-jzshj" pod "c344af02-b407-421d-aea3-6f3789ee9017" (UID: "c344af02-b407-421d-aea3-6f3789ee9017"). InnerVolumeSpecName "kube-api-access-jzshj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:00.626840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:00.626802 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c344af02-b407-421d-aea3-6f3789ee9017-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:00.626840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:00.626834 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzshj\" (UniqueName: \"kubernetes.io/projected/c344af02-b407-421d-aea3-6f3789ee9017-kube-api-access-jzshj\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:00.626840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:00.626845 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c344af02-b407-421d-aea3-6f3789ee9017-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:00.740198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:00.740161 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Oct 30 21:55:00.740396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:00.740218 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" podUID="3bf71b99-a679-4d29-b081-5d26c7abfcde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Oct 30 21:55:01.234206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.234179 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk" Oct 30 21:55:01.234377 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.234213 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk" event={"ID":"c344af02-b407-421d-aea3-6f3789ee9017","Type":"ContainerDied","Data":"b04f41d039916c960fa680251b3849beff51f8b70df175ec1ba13240ec2b9f82"} Oct 30 21:55:01.234377 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.234265 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b04f41d039916c960fa680251b3849beff51f8b70df175ec1ba13240ec2b9f82" Oct 30 21:55:01.560088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.559806 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:55:01.609268 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.609174 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9"] Oct 30 21:55:01.615240 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.615206 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l"] Oct 30 21:55:01.619990 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.619957 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh"] Oct 30 21:55:01.641807 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.641766 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-workload-socket\") pod \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " Oct 30 21:55:01.641990 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.641875 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-kube-api-access-zblm9\" (UniqueName: \"kubernetes.io/projected/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-kube-api-access-zblm9\") pod \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " Oct 30 21:55:01.642099 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.641998 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-podinfo\") pod \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " Oct 30 21:55:01.642099 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.642072 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-data\") pod \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " Oct 30 21:55:01.642229 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.642128 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-credential-socket\") pod \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " Oct 30 21:55:01.642229 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.642167 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-token\" (UniqueName: \"kubernetes.io/projected/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-token\") pod \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " Oct 30 21:55:01.642229 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.642206 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-envoy\") pod \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " Oct 30 21:55:01.642450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.642244 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-workload-certs\") pod \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " Oct 30 21:55:01.642450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.642291 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-workload-socket" pod "d3d50005-aee2-4d9f-b966-7f8a41df5a5a" (UID: "d3d50005-aee2-4d9f-b966-7f8a41df5a5a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:01.642450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.642338 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istiod-ca-cert\") pod \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\" (UID: \"d3d50005-aee2-4d9f-b966-7f8a41df5a5a\") " Oct 30 21:55:01.642622 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.642495 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-data" pod "d3d50005-aee2-4d9f-b966-7f8a41df5a5a" (UID: "d3d50005-aee2-4d9f-b966-7f8a41df5a5a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:01.643075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.642926 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:01.643075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.642952 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:01.644275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.643353 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istiod-ca-cert" pod "d3d50005-aee2-4d9f-b966-7f8a41df5a5a" (UID: "d3d50005-aee2-4d9f-b966-7f8a41df5a5a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:01.644275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.643521 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-workload-certs" pod "d3d50005-aee2-4d9f-b966-7f8a41df5a5a" (UID: "d3d50005-aee2-4d9f-b966-7f8a41df5a5a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:01.644275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.643798 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-credential-socket" pod "d3d50005-aee2-4d9f-b966-7f8a41df5a5a" (UID: "d3d50005-aee2-4d9f-b966-7f8a41df5a5a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:01.645211 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.645174 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-kube-api-access-zblm9" pod "d3d50005-aee2-4d9f-b966-7f8a41df5a5a" (UID: "d3d50005-aee2-4d9f-b966-7f8a41df5a5a"). InnerVolumeSpecName "kube-api-access-zblm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:01.645905 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.645883 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-envoy" pod "d3d50005-aee2-4d9f-b966-7f8a41df5a5a" (UID: "d3d50005-aee2-4d9f-b966-7f8a41df5a5a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:01.646912 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.646881 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-podinfo" pod "d3d50005-aee2-4d9f-b966-7f8a41df5a5a" (UID: "d3d50005-aee2-4d9f-b966-7f8a41df5a5a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:55:01.647218 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.647194 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-token" pod "d3d50005-aee2-4d9f-b966-7f8a41df5a5a" (UID: "d3d50005-aee2-4d9f-b966-7f8a41df5a5a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:01.739685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.739650 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Oct 30 21:55:01.739870 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.739700 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" podUID="3bf71b99-a679-4d29-b081-5d26c7abfcde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Oct 30 21:55:01.744952 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.744919 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zblm9\" (UniqueName: \"kubernetes.io/projected/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-kube-api-access-zblm9\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:01.745061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.744959 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:01.745061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.744975 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:01.745061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.744988 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:01.745061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.745001 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:01.745061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.745012 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:01.745061 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:01.745024 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3d50005-aee2-4d9f-b966-7f8a41df5a5a-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:02.245563 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:02.245518 4926 generic.go:356] "Generic (PLEG): container finished" podID="d3d50005-aee2-4d9f-b966-7f8a41df5a5a" containerID="3c6f0544f64cd19b22520827eb6bb494235131d7ad88350d3a6cf3bb4c09312f" exitCode=0 Oct 30 21:55:02.245563 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:02.245551 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" Oct 30 21:55:02.245842 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:02.245603 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" event={"ID":"d3d50005-aee2-4d9f-b966-7f8a41df5a5a","Type":"ContainerDied","Data":"3c6f0544f64cd19b22520827eb6bb494235131d7ad88350d3a6cf3bb4c09312f"} Oct 30 21:55:02.245842 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:02.245677 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp" event={"ID":"d3d50005-aee2-4d9f-b966-7f8a41df5a5a","Type":"ContainerDied","Data":"b6a86c3d3ccf58e623d2536bdd24ce07cd9bbb31ff2ac6c3def9e0a94107fa50"} Oct 30 21:55:02.246123 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:02.246103 4926 scope.go:122] "RemoveContainer" containerID="3c6f0544f64cd19b22520827eb6bb494235131d7ad88350d3a6cf3bb4c09312f" Oct 30 21:55:02.263993 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:02.263965 4926 scope.go:122] "RemoveContainer" containerID="0df3de27d0427d80095de282a66195297d01ea84ad9d5dd34536958738e74a1d" Oct 30 21:55:02.308697 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:02.308655 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp"] Oct 30 21:55:02.309116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:02.309020 4926 scope.go:122] "RemoveContainer" containerID="3c6f0544f64cd19b22520827eb6bb494235131d7ad88350d3a6cf3bb4c09312f" Oct 30 21:55:02.309669 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:55:02.309646 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c6f0544f64cd19b22520827eb6bb494235131d7ad88350d3a6cf3bb4c09312f\": container with ID starting with 3c6f0544f64cd19b22520827eb6bb494235131d7ad88350d3a6cf3bb4c09312f not found: ID does not exist" containerID="3c6f0544f64cd19b22520827eb6bb494235131d7ad88350d3a6cf3bb4c09312f" Oct 30 21:55:02.309769 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:02.309677 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c6f0544f64cd19b22520827eb6bb494235131d7ad88350d3a6cf3bb4c09312f"} err="failed to get container status \"3c6f0544f64cd19b22520827eb6bb494235131d7ad88350d3a6cf3bb4c09312f\": rpc error: code = NotFound desc = could not find container \"3c6f0544f64cd19b22520827eb6bb494235131d7ad88350d3a6cf3bb4c09312f\": container with ID starting with 3c6f0544f64cd19b22520827eb6bb494235131d7ad88350d3a6cf3bb4c09312f not found: ID does not exist" Oct 30 21:55:02.309769 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:02.309694 4926 scope.go:122] "RemoveContainer" containerID="0df3de27d0427d80095de282a66195297d01ea84ad9d5dd34536958738e74a1d" Oct 30 21:55:02.310010 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:55:02.309977 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0df3de27d0427d80095de282a66195297d01ea84ad9d5dd34536958738e74a1d\": container with ID starting with 0df3de27d0427d80095de282a66195297d01ea84ad9d5dd34536958738e74a1d not found: ID does not exist" containerID="0df3de27d0427d80095de282a66195297d01ea84ad9d5dd34536958738e74a1d" Oct 30 21:55:02.310098 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:02.310009 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df3de27d0427d80095de282a66195297d01ea84ad9d5dd34536958738e74a1d"} err="failed to get container status \"0df3de27d0427d80095de282a66195297d01ea84ad9d5dd34536958738e74a1d\": rpc error: code = NotFound desc = could not find container \"0df3de27d0427d80095de282a66195297d01ea84ad9d5dd34536958738e74a1d\": container with ID starting with 0df3de27d0427d80095de282a66195297d01ea84ad9d5dd34536958738e74a1d not found: ID does not exist" Oct 30 21:55:02.319759 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:02.319727 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-70d3a9-openshift-default-6cb8bc6bf5-qnrdp"] Oct 30 21:55:02.740025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:02.739988 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Oct 30 21:55:02.740578 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:02.740072 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" podUID="3bf71b99-a679-4d29-b081-5d26c7abfcde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Oct 30 21:55:03.019773 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:03.019680 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3d50005-aee2-4d9f-b966-7f8a41df5a5a" path="/var/lib/kubelet/pods/d3d50005-aee2-4d9f-b966-7f8a41df5a5a/volumes" Oct 30 21:55:03.739913 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:03.739874 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" start-of-body= Oct 30 21:55:03.740141 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:03.739927 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" podUID="3bf71b99-a679-4d29-b081-5d26c7abfcde" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.42:15021/healthz/ready\": dial tcp 10.132.1.42:15021: connect: connection refused" Oct 30 21:55:04.812103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:04.811376 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:55:04.812694 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:04.812329 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:55:04.815789 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:04.815759 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:55:05.415973 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:05.415924 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f"] Oct 30 21:55:05.421224 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:05.421193 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-66cc54-6ps4f"] Oct 30 21:55:06.044824 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:06.044781 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv" podUID="1210d99e-e106-4c6e-8947-cb54d3ec4c2e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:55:07.017566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:07.017533 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="398b8d42-6bcc-4a61-8807-05fbf6d9df47" path="/var/lib/kubelet/pods/398b8d42-6bcc-4a61-8807-05fbf6d9df47/volumes" Oct 30 21:55:07.496877 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:55:07.496800 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:55:08.579103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:08.579070 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:55:10.147083 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:10.147004 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44"] Oct 30 21:55:10.147846 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:10.147766 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" podUID="50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" containerName="istio-proxy" containerID="cri-o://39b11732a64b796d59ccc618e70c6e4e1f4c6cad62f1ca620736ae58ec607468" gracePeriod=30 Oct 30 21:55:10.765271 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:10.765229 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v"] Oct 30 21:55:10.766149 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:10.766128 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d3d50005-aee2-4d9f-b966-7f8a41df5a5a" containerName="istio-proxy" Oct 30 21:55:10.766248 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:10.766151 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d50005-aee2-4d9f-b966-7f8a41df5a5a" containerName="istio-proxy" Oct 30 21:55:10.766248 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:10.766182 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c344af02-b407-421d-aea3-6f3789ee9017" containerName="revoke-keys" Oct 30 21:55:10.766248 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:10.766188 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c344af02-b407-421d-aea3-6f3789ee9017" containerName="revoke-keys" Oct 30 21:55:10.766414 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:10.766328 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d50005-aee2-4d9f-b966-7f8a41df5a5a" containerName="istio-proxy" Oct 30 21:55:10.766414 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:10.766342 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c344af02-b407-421d-aea3-6f3789ee9017" containerName="revoke-keys" Oct 30 21:55:10.772135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:10.772112 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v" Oct 30 21:55:10.788596 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:10.788573 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:55:10.789302 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:10.789279 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:55:10.811767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:10.811733 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v"] Oct 30 21:55:10.872387 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:10.872346 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc62639f-743f-40a1-ae44-d7c132e74fb6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v\" (UID: \"bc62639f-743f-40a1-ae44-d7c132e74fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v" Oct 30 21:55:10.872610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:10.872552 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc62639f-743f-40a1-ae44-d7c132e74fb6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v\" (UID: \"bc62639f-743f-40a1-ae44-d7c132e74fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v" Oct 30 21:55:10.872685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:10.872605 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klrxz\" (UniqueName: \"kubernetes.io/projected/bc62639f-743f-40a1-ae44-d7c132e74fb6-kube-api-access-klrxz\") pod \"maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v\" (UID: \"bc62639f-743f-40a1-ae44-d7c132e74fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v" Oct 30 21:55:10.974071 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:10.974020 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc62639f-743f-40a1-ae44-d7c132e74fb6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v\" (UID: \"bc62639f-743f-40a1-ae44-d7c132e74fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v" Oct 30 21:55:10.974313 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:10.974290 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc62639f-743f-40a1-ae44-d7c132e74fb6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v\" (UID: \"bc62639f-743f-40a1-ae44-d7c132e74fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v" Oct 30 21:55:10.974391 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:10.974376 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klrxz\" (UniqueName: \"kubernetes.io/projected/bc62639f-743f-40a1-ae44-d7c132e74fb6-kube-api-access-klrxz\") pod \"maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v\" (UID: \"bc62639f-743f-40a1-ae44-d7c132e74fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v" Oct 30 21:55:10.974883 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:10.974858 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc62639f-743f-40a1-ae44-d7c132e74fb6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v\" (UID: \"bc62639f-743f-40a1-ae44-d7c132e74fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v" Oct 30 21:55:10.977504 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:10.977474 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc62639f-743f-40a1-ae44-d7c132e74fb6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v\" (UID: \"bc62639f-743f-40a1-ae44-d7c132e74fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v" Oct 30 21:55:11.003585 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:11.003548 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klrxz\" (UniqueName: \"kubernetes.io/projected/bc62639f-743f-40a1-ae44-d7c132e74fb6-kube-api-access-klrxz\") pod \"maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v\" (UID: \"bc62639f-743f-40a1-ae44-d7c132e74fb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v" Oct 30 21:55:11.090344 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:11.090267 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v" Oct 30 21:55:11.423825 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:11.423792 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v"] Oct 30 21:55:11.424648 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:55:11.424619 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc62639f_743f_40a1_ae44_d7c132e74fb6.slice/crio-a98a387019979c3d3b5ee0da2ba9507384b85236f78708da851e040c09c8cab2 WatchSource:0}: Error finding container a98a387019979c3d3b5ee0da2ba9507384b85236f78708da851e040c09c8cab2: Status 404 returned error can't find the container with id a98a387019979c3d3b5ee0da2ba9507384b85236f78708da851e040c09c8cab2 Oct 30 21:55:12.381899 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:12.381871 4926 generic.go:356] "Generic (PLEG): container finished" podID="bc62639f-743f-40a1-ae44-d7c132e74fb6" containerID="2ce1d6f5386247b0be5e6a7fc463ac02fdd0487f9a29bcb4c067bec775fe335f" exitCode=0 Oct 30 21:55:12.382542 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:12.382517 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v" event={"ID":"bc62639f-743f-40a1-ae44-d7c132e74fb6","Type":"ContainerDied","Data":"2ce1d6f5386247b0be5e6a7fc463ac02fdd0487f9a29bcb4c067bec775fe335f"} Oct 30 21:55:12.382672 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:12.382572 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v" event={"ID":"bc62639f-743f-40a1-ae44-d7c132e74fb6","Type":"ContainerStarted","Data":"a98a387019979c3d3b5ee0da2ba9507384b85236f78708da851e040c09c8cab2"} Oct 30 21:55:12.384851 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:12.384829 4926 generic.go:356] "Generic (PLEG): container finished" podID="1210d99e-e106-4c6e-8947-cb54d3ec4c2e" containerID="22714bbf960eae8afff4887f37d9d26b2306f61e8f15f438c652665d2c7f8b97" exitCode=0 Oct 30 21:55:12.384966 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:12.384903 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv" event={"ID":"1210d99e-e106-4c6e-8947-cb54d3ec4c2e","Type":"ContainerDied","Data":"22714bbf960eae8afff4887f37d9d26b2306f61e8f15f438c652665d2c7f8b97"} Oct 30 21:55:12.549102 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:12.549066 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv" Oct 30 21:55:12.598556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:12.598523 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1210d99e-e106-4c6e-8947-cb54d3ec4c2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1210d99e-e106-4c6e-8947-cb54d3ec4c2e-plugins-config-volume\") pod \"1210d99e-e106-4c6e-8947-cb54d3ec4c2e\" (UID: \"1210d99e-e106-4c6e-8947-cb54d3ec4c2e\") " Oct 30 21:55:12.598778 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:12.598611 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1210d99e-e106-4c6e-8947-cb54d3ec4c2e-kube-api-access-zkqhb\" (UniqueName: \"kubernetes.io/projected/1210d99e-e106-4c6e-8947-cb54d3ec4c2e-kube-api-access-zkqhb\") pod \"1210d99e-e106-4c6e-8947-cb54d3ec4c2e\" (UID: \"1210d99e-e106-4c6e-8947-cb54d3ec4c2e\") " Oct 30 21:55:12.599014 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:12.598987 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1210d99e-e106-4c6e-8947-cb54d3ec4c2e-plugins-config-volume" pod "1210d99e-e106-4c6e-8947-cb54d3ec4c2e" (UID: "1210d99e-e106-4c6e-8947-cb54d3ec4c2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:12.599168 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:12.599152 4926 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1210d99e-e106-4c6e-8947-cb54d3ec4c2e-plugins-config-volume\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:12.600932 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:12.600906 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1210d99e-e106-4c6e-8947-cb54d3ec4c2e-kube-api-access-zkqhb" pod "1210d99e-e106-4c6e-8947-cb54d3ec4c2e" (UID: "1210d99e-e106-4c6e-8947-cb54d3ec4c2e"). InnerVolumeSpecName "kube-api-access-zkqhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:12.700754 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:12.700709 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkqhb\" (UniqueName: \"kubernetes.io/projected/1210d99e-e106-4c6e-8947-cb54d3ec4c2e-kube-api-access-zkqhb\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:13.397913 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:13.397807 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv" Oct 30 21:55:13.397913 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:13.397817 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv" event={"ID":"1210d99e-e106-4c6e-8947-cb54d3ec4c2e","Type":"ContainerDied","Data":"6b95eb33dd68ff828bd60c79d1c7c4fee4686f936f1f1d6607ba93b6e75d85ff"} Oct 30 21:55:13.398269 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:13.397954 4926 scope.go:122] "RemoveContainer" containerID="22714bbf960eae8afff4887f37d9d26b2306f61e8f15f438c652665d2c7f8b97" Oct 30 21:55:13.454576 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:13.454524 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv"] Oct 30 21:55:13.761616 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:13.750621 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-f5bef0-55dd7bcb78-8v6lv"] Oct 30 21:55:13.846070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:13.845886 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v" Oct 30 21:55:13.869268 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:13.869238 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:55:13.870233 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:13.870197 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 21:55:13.952373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:13.952332 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc62639f-743f-40a1-ae44-d7c132e74fb6-kube-api-access-klrxz\" (UniqueName: \"kubernetes.io/projected/bc62639f-743f-40a1-ae44-d7c132e74fb6-kube-api-access-klrxz\") pod \"bc62639f-743f-40a1-ae44-d7c132e74fb6\" (UID: \"bc62639f-743f-40a1-ae44-d7c132e74fb6\") " Oct 30 21:55:13.952608 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:13.952492 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc62639f-743f-40a1-ae44-d7c132e74fb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc62639f-743f-40a1-ae44-d7c132e74fb6-maas-api-xks-ca\") pod \"bc62639f-743f-40a1-ae44-d7c132e74fb6\" (UID: \"bc62639f-743f-40a1-ae44-d7c132e74fb6\") " Oct 30 21:55:13.952822 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:13.952703 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc62639f-743f-40a1-ae44-d7c132e74fb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc62639f-743f-40a1-ae44-d7c132e74fb6-maas-api-openshift-ca\") pod \"bc62639f-743f-40a1-ae44-d7c132e74fb6\" (UID: \"bc62639f-743f-40a1-ae44-d7c132e74fb6\") " Oct 30 21:55:13.953206 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:13.953165 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc62639f-743f-40a1-ae44-d7c132e74fb6-maas-api-openshift-ca" pod "bc62639f-743f-40a1-ae44-d7c132e74fb6" (UID: "bc62639f-743f-40a1-ae44-d7c132e74fb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:13.953374 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:13.953345 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc62639f-743f-40a1-ae44-d7c132e74fb6-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:13.954957 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:13.954932 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc62639f-743f-40a1-ae44-d7c132e74fb6-kube-api-access-klrxz" pod "bc62639f-743f-40a1-ae44-d7c132e74fb6" (UID: "bc62639f-743f-40a1-ae44-d7c132e74fb6"). InnerVolumeSpecName "kube-api-access-klrxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:13.955611 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:13.955588 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc62639f-743f-40a1-ae44-d7c132e74fb6-maas-api-xks-ca" pod "bc62639f-743f-40a1-ae44-d7c132e74fb6" (UID: "bc62639f-743f-40a1-ae44-d7c132e74fb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:14.035813 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.035714 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76"] Oct 30 21:55:14.041482 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.041447 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f5bef0-vjm76"] Oct 30 21:55:14.054896 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.054855 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc62639f-743f-40a1-ae44-d7c132e74fb6-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:14.054896 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.054885 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klrxz\" (UniqueName: \"kubernetes.io/projected/bc62639f-743f-40a1-ae44-d7c132e74fb6-kube-api-access-klrxz\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:14.139279 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.139221 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm"] Oct 30 21:55:14.139541 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.139481 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" podUID="7c6202f3-5168-4b2a-a167-c87f1fa98fd1" containerName="istio-proxy" containerID="cri-o://0d914076ebb52e1a0d9e902bdead09db6e43f9b9f855dca5b8a131d1584f6026" gracePeriod=30 Oct 30 21:55:14.415837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.415801 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v" Oct 30 21:55:14.416063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.415833 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v" event={"ID":"bc62639f-743f-40a1-ae44-d7c132e74fb6","Type":"ContainerDied","Data":"a98a387019979c3d3b5ee0da2ba9507384b85236f78708da851e040c09c8cab2"} Oct 30 21:55:14.416063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.415885 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a98a387019979c3d3b5ee0da2ba9507384b85236f78708da851e040c09c8cab2" Oct 30 21:55:14.727360 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.727281 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s"] Oct 30 21:55:14.727880 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.727865 4926 cpu_manager.go:414] "containerMap: removing container" podUID="1210d99e-e106-4c6e-8947-cb54d3ec4c2e" containerName="payload-processing" Oct 30 21:55:14.727939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.727881 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1210d99e-e106-4c6e-8947-cb54d3ec4c2e" containerName="payload-processing" Oct 30 21:55:14.727939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.727908 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d3d50005-aee2-4d9f-b966-7f8a41df5a5a" containerName="istio-proxy" Oct 30 21:55:14.727939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.727914 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3d50005-aee2-4d9f-b966-7f8a41df5a5a" containerName="istio-proxy" Oct 30 21:55:14.727939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.727930 4926 cpu_manager.go:414] "containerMap: removing container" podUID="bc62639f-743f-40a1-ae44-d7c132e74fb6" containerName="revoke-keys" Oct 30 21:55:14.727939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.727935 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc62639f-743f-40a1-ae44-d7c132e74fb6" containerName="revoke-keys" Oct 30 21:55:14.728158 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.728057 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3d50005-aee2-4d9f-b966-7f8a41df5a5a" containerName="istio-proxy" Oct 30 21:55:14.728158 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.728066 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc62639f-743f-40a1-ae44-d7c132e74fb6" containerName="revoke-keys" Oct 30 21:55:14.728158 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.728077 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="1210d99e-e106-4c6e-8947-cb54d3ec4c2e" containerName="payload-processing" Oct 30 21:55:14.731827 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.731810 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s" Oct 30 21:55:14.751543 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.751515 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s"] Oct 30 21:55:14.757767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.757737 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:55:14.757928 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.757828 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:55:14.876347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.876308 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1cde02-c2dd-4dfb-8596-ba673bfcb5ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s\" (UID: \"af1cde02-c2dd-4dfb-8596-ba673bfcb5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s" Oct 30 21:55:14.876793 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.876377 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqrtm\" (UniqueName: \"kubernetes.io/projected/af1cde02-c2dd-4dfb-8596-ba673bfcb5ec-kube-api-access-rqrtm\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s\" (UID: \"af1cde02-c2dd-4dfb-8596-ba673bfcb5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s" Oct 30 21:55:14.876793 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.876460 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1cde02-c2dd-4dfb-8596-ba673bfcb5ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s\" (UID: \"af1cde02-c2dd-4dfb-8596-ba673bfcb5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s" Oct 30 21:55:14.978020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.977927 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1cde02-c2dd-4dfb-8596-ba673bfcb5ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s\" (UID: \"af1cde02-c2dd-4dfb-8596-ba673bfcb5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s" Oct 30 21:55:14.978020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.977988 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqrtm\" (UniqueName: \"kubernetes.io/projected/af1cde02-c2dd-4dfb-8596-ba673bfcb5ec-kube-api-access-rqrtm\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s\" (UID: \"af1cde02-c2dd-4dfb-8596-ba673bfcb5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s" Oct 30 21:55:14.978020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.978009 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1cde02-c2dd-4dfb-8596-ba673bfcb5ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s\" (UID: \"af1cde02-c2dd-4dfb-8596-ba673bfcb5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s" Oct 30 21:55:14.978887 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.978856 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1cde02-c2dd-4dfb-8596-ba673bfcb5ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s\" (UID: \"af1cde02-c2dd-4dfb-8596-ba673bfcb5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s" Oct 30 21:55:14.980619 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.980587 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1cde02-c2dd-4dfb-8596-ba673bfcb5ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s\" (UID: \"af1cde02-c2dd-4dfb-8596-ba673bfcb5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s" Oct 30 21:55:14.998026 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:14.997992 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqrtm\" (UniqueName: \"kubernetes.io/projected/af1cde02-c2dd-4dfb-8596-ba673bfcb5ec-kube-api-access-rqrtm\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s\" (UID: \"af1cde02-c2dd-4dfb-8596-ba673bfcb5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s" Oct 30 21:55:15.021082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.021041 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1210d99e-e106-4c6e-8947-cb54d3ec4c2e" path="/var/lib/kubelet/pods/1210d99e-e106-4c6e-8947-cb54d3ec4c2e/volumes" Oct 30 21:55:15.021777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.021757 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b32b9802-d274-43ca-a973-1a17b0dfcc72" path="/var/lib/kubelet/pods/b32b9802-d274-43ca-a973-1a17b0dfcc72/volumes" Oct 30 21:55:15.046899 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.046863 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s" Oct 30 21:55:15.388902 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:55:15.388863 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1cde02_c2dd_4dfb_8596_ba673bfcb5ec.slice/crio-65eeb786593a7753572302f0af02fb04cb2c8ffd0cc0af095d71f194afe24b01 WatchSource:0}: Error finding container 65eeb786593a7753572302f0af02fb04cb2c8ffd0cc0af095d71f194afe24b01: Status 404 returned error can't find the container with id 65eeb786593a7753572302f0af02fb04cb2c8ffd0cc0af095d71f194afe24b01 Oct 30 21:55:15.389300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.389274 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s"] Oct 30 21:55:15.429895 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.429855 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s" event={"ID":"af1cde02-c2dd-4dfb-8596-ba673bfcb5ec","Type":"ContainerStarted","Data":"65eeb786593a7753572302f0af02fb04cb2c8ffd0cc0af095d71f194afe24b01"} Oct 30 21:55:15.432811 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.432779 4926 generic.go:356] "Generic (PLEG): container finished" podID="50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" containerID="39b11732a64b796d59ccc618e70c6e4e1f4c6cad62f1ca620736ae58ec607468" exitCode=0 Oct 30 21:55:15.432956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.432847 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" event={"ID":"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5","Type":"ContainerDied","Data":"39b11732a64b796d59ccc618e70c6e4e1f4c6cad62f1ca620736ae58ec607468"} Oct 30 21:55:15.432956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.432911 4926 scope.go:122] "RemoveContainer" containerID="2043dd192beaa7e842e844f63a0681d3dee2910e3ca5150a364a7e4f4a2853fb" Oct 30 21:55:15.441176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.441155 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:55:15.606647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.598490 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-workload-socket\") pod \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " Oct 30 21:55:15.606647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.598627 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-credential-socket\") pod \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " Oct 30 21:55:15.606647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.598783 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-workload-certs\") pod \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " Oct 30 21:55:15.606647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.598814 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-data\") pod \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " Oct 30 21:55:15.606647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.598850 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istiod-ca-cert\") pod \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " Oct 30 21:55:15.606647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.598933 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-podinfo\") pod \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " Oct 30 21:55:15.606647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.598965 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-kube-api-access-psg9n\" (UniqueName: \"kubernetes.io/projected/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-kube-api-access-psg9n\") pod \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " Oct 30 21:55:15.606647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.599020 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-token\" (UniqueName: \"kubernetes.io/projected/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-token\") pod \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " Oct 30 21:55:15.606647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.599066 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-envoy\") pod \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\" (UID: \"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5\") " Oct 30 21:55:15.607318 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.606944 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-credential-socket" pod "50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" (UID: "50bd09f8-bbc4-41f3-ae8f-f517e50a78b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:15.607318 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.607009 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-workload-certs" pod "50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" (UID: "50bd09f8-bbc4-41f3-ae8f-f517e50a78b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:15.607318 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.607152 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-workload-socket" pod "50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" (UID: "50bd09f8-bbc4-41f3-ae8f-f517e50a78b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:15.607481 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.607391 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istiod-ca-cert" pod "50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" (UID: "50bd09f8-bbc4-41f3-ae8f-f517e50a78b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:15.607481 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.607400 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-data" pod "50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" (UID: "50bd09f8-bbc4-41f3-ae8f-f517e50a78b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:15.610317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.609516 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-podinfo" pod "50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" (UID: "50bd09f8-bbc4-41f3-ae8f-f517e50a78b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:55:15.610317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.609639 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-envoy" pod "50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" (UID: "50bd09f8-bbc4-41f3-ae8f-f517e50a78b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:15.610317 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.609704 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-token" pod "50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" (UID: "50bd09f8-bbc4-41f3-ae8f-f517e50a78b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:15.611179 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.611155 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-kube-api-access-psg9n" pod "50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" (UID: "50bd09f8-bbc4-41f3-ae8f-f517e50a78b5"). InnerVolumeSpecName "kube-api-access-psg9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:15.700732 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.700700 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:15.700732 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.700729 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:15.700732 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.700738 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:15.700942 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.700746 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:15.700942 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.700754 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:15.700942 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.700762 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:15.700942 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.700770 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psg9n\" (UniqueName: \"kubernetes.io/projected/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-kube-api-access-psg9n\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:15.700942 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.700778 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:15.700942 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:15.700786 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:16.444620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:16.444586 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" Oct 30 21:55:16.445165 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:16.444621 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44" event={"ID":"50bd09f8-bbc4-41f3-ae8f-f517e50a78b5","Type":"ContainerDied","Data":"9db9ef35a5e485e1fa3b979a31629be7c3306146419b8e4ad33dfe0ed128a79b"} Oct 30 21:55:16.445165 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:16.444694 4926 scope.go:122] "RemoveContainer" containerID="39b11732a64b796d59ccc618e70c6e4e1f4c6cad62f1ca620736ae58ec607468" Oct 30 21:55:16.447081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:16.447025 4926 generic.go:356] "Generic (PLEG): container finished" podID="af1cde02-c2dd-4dfb-8596-ba673bfcb5ec" containerID="56db70cd8511776a651eeb315d5c8748db5668f525f583b21010d115befcc70d" exitCode=0 Oct 30 21:55:16.447214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:16.447146 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s" event={"ID":"af1cde02-c2dd-4dfb-8596-ba673bfcb5ec","Type":"ContainerDied","Data":"56db70cd8511776a651eeb315d5c8748db5668f525f583b21010d115befcc70d"} Oct 30 21:55:16.539150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:16.539110 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44"] Oct 30 21:55:16.548645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:16.548608 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-203209-openshift-default-6f6fdc6799-bkx44"] Oct 30 21:55:16.638214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:16.638105 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" containerID="cri-o://db69411cdd265effbe3d89419b224b951d6f8e77a09f653110e2ffb61d69aa1f" gracePeriod=45 Oct 30 21:55:16.640642 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:16.640600 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" containerID="cri-o://42dc88d0b40943b1564b022a0be121c14617227799ecb0751d2e296433e0d279" gracePeriod=45 Oct 30 21:55:16.646130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:16.645995 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" containerID="cri-o://8c065bf7415ba4bd99b83fde94989ae6a27de60325cc340d5d781f6ba5aa8a59" gracePeriod=45 Oct 30 21:55:16.694342 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:16.694301 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": dial tcp 10.132.0.66:8000: connect: connection refused" Oct 30 21:55:16.988976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:16.988553 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:55:17.030381 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.027766 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" path="/var/lib/kubelet/pods/50bd09f8-bbc4-41f3-ae8f-f517e50a78b5/volumes" Oct 30 21:55:17.046895 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.046861 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:55:17.048956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.048930 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:55:17.122083 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.122025 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-kube-api-access-wz7lv\" (UniqueName: \"kubernetes.io/projected/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-kube-api-access-wz7lv\") pod \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " Oct 30 21:55:17.122266 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.122152 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-model-cache\") pod \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " Oct 30 21:55:17.122266 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.122204 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-tmp-dir\") pod \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " Oct 30 21:55:17.122478 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.122450 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-model-cache" pod "c6f1e2cb-36a7-435d-b61e-8ab3272c591a" (UID: "c6f1e2cb-36a7-435d-b61e-8ab3272c591a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:17.122478 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.122470 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-dshm\") pod \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " Oct 30 21:55:17.122571 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.122476 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-tmp-dir" pod "c6f1e2cb-36a7-435d-b61e-8ab3272c591a" (UID: "c6f1e2cb-36a7-435d-b61e-8ab3272c591a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:17.122571 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.122515 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-tls-certs\") pod \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " Oct 30 21:55:17.122571 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.122561 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-home\" (UniqueName: \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-home\") pod \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\" (UID: \"c6f1e2cb-36a7-435d-b61e-8ab3272c591a\") " Oct 30 21:55:17.122910 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.122839 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-home" pod "c6f1e2cb-36a7-435d-b61e-8ab3272c591a" (UID: "c6f1e2cb-36a7-435d-b61e-8ab3272c591a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:17.123265 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.123240 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:17.123356 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.123269 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:17.123356 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.123311 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:17.124797 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.124769 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-tls-certs" pod "c6f1e2cb-36a7-435d-b61e-8ab3272c591a" (UID: "c6f1e2cb-36a7-435d-b61e-8ab3272c591a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:17.124907 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.124796 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-dshm" pod "c6f1e2cb-36a7-435d-b61e-8ab3272c591a" (UID: "c6f1e2cb-36a7-435d-b61e-8ab3272c591a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:17.124970 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.124931 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-kube-api-access-wz7lv" pod "c6f1e2cb-36a7-435d-b61e-8ab3272c591a" (UID: "c6f1e2cb-36a7-435d-b61e-8ab3272c591a"). InnerVolumeSpecName "kube-api-access-wz7lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:17.224617 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.224515 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-tmp-dir\") pod \"e190db51-e133-4ca2-a26b-f89c2b67da45\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " Oct 30 21:55:17.224617 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.224587 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-tmp-dir\") pod \"cd3973a3-874a-4e67-a809-60b41b853435\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " Oct 30 21:55:17.224896 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.224687 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-dshm\") pod \"e190db51-e133-4ca2-a26b-f89c2b67da45\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " Oct 30 21:55:17.224896 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.224749 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e190db51-e133-4ca2-a26b-f89c2b67da45-kube-api-access-8c98r\" (UniqueName: \"kubernetes.io/projected/e190db51-e133-4ca2-a26b-f89c2b67da45-kube-api-access-8c98r\") pod \"e190db51-e133-4ca2-a26b-f89c2b67da45\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " Oct 30 21:55:17.225026 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.224952 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e190db51-e133-4ca2-a26b-f89c2b67da45-tls-certs\" (UniqueName: \"kubernetes.io/secret/e190db51-e133-4ca2-a26b-f89c2b67da45-tls-certs\") pod \"e190db51-e133-4ca2-a26b-f89c2b67da45\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " Oct 30 21:55:17.225026 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.224994 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-model-cache\") pod \"e190db51-e133-4ca2-a26b-f89c2b67da45\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " Oct 30 21:55:17.225180 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.225025 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-home\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-home\") pod \"cd3973a3-874a-4e67-a809-60b41b853435\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " Oct 30 21:55:17.225180 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.225073 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-home\" (UniqueName: \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-home\") pod \"e190db51-e133-4ca2-a26b-f89c2b67da45\" (UID: \"e190db51-e133-4ca2-a26b-f89c2b67da45\") " Oct 30 21:55:17.225180 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.225106 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd3973a3-874a-4e67-a809-60b41b853435-tls-certs\" (UniqueName: \"kubernetes.io/secret/cd3973a3-874a-4e67-a809-60b41b853435-tls-certs\") pod \"cd3973a3-874a-4e67-a809-60b41b853435\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " Oct 30 21:55:17.225180 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.225134 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-tmp-dir" pod "cd3973a3-874a-4e67-a809-60b41b853435" (UID: "cd3973a3-874a-4e67-a809-60b41b853435"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:17.225180 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.225162 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-dshm\") pod \"cd3973a3-874a-4e67-a809-60b41b853435\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " Oct 30 21:55:17.225455 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.225297 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd3973a3-874a-4e67-a809-60b41b853435-kube-api-access-9rctr\" (UniqueName: \"kubernetes.io/projected/cd3973a3-874a-4e67-a809-60b41b853435-kube-api-access-9rctr\") pod \"cd3973a3-874a-4e67-a809-60b41b853435\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " Oct 30 21:55:17.225455 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.225379 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-model-cache\") pod \"cd3973a3-874a-4e67-a809-60b41b853435\" (UID: \"cd3973a3-874a-4e67-a809-60b41b853435\") " Oct 30 21:55:17.225576 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.225473 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-model-cache" pod "e190db51-e133-4ca2-a26b-f89c2b67da45" (UID: "e190db51-e133-4ca2-a26b-f89c2b67da45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:17.225916 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.225877 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-home" pod "e190db51-e133-4ca2-a26b-f89c2b67da45" (UID: "e190db51-e133-4ca2-a26b-f89c2b67da45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:17.226095 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.225987 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-home" pod "cd3973a3-874a-4e67-a809-60b41b853435" (UID: "cd3973a3-874a-4e67-a809-60b41b853435"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:17.226469 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.226312 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-model-cache" pod "cd3973a3-874a-4e67-a809-60b41b853435" (UID: "cd3973a3-874a-4e67-a809-60b41b853435"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:17.226621 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.226361 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-tmp-dir" pod "e190db51-e133-4ca2-a26b-f89c2b67da45" (UID: "e190db51-e133-4ca2-a26b-f89c2b67da45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:17.226808 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.226115 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:17.226922 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.226908 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:17.227018 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.227007 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:17.227145 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.227128 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz7lv\" (UniqueName: \"kubernetes.io/projected/c6f1e2cb-36a7-435d-b61e-8ab3272c591a-kube-api-access-wz7lv\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:17.227241 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.227228 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:17.229148 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.228361 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-dshm" pod "cd3973a3-874a-4e67-a809-60b41b853435" (UID: "cd3973a3-874a-4e67-a809-60b41b853435"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:17.231221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.230196 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e190db51-e133-4ca2-a26b-f89c2b67da45-kube-api-access-8c98r" pod "e190db51-e133-4ca2-a26b-f89c2b67da45" (UID: "e190db51-e133-4ca2-a26b-f89c2b67da45"). InnerVolumeSpecName "kube-api-access-8c98r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:17.231221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.230421 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd3973a3-874a-4e67-a809-60b41b853435-tls-certs" pod "cd3973a3-874a-4e67-a809-60b41b853435" (UID: "cd3973a3-874a-4e67-a809-60b41b853435"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:17.231221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.230780 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd3973a3-874a-4e67-a809-60b41b853435-kube-api-access-9rctr" pod "cd3973a3-874a-4e67-a809-60b41b853435" (UID: "cd3973a3-874a-4e67-a809-60b41b853435"). InnerVolumeSpecName "kube-api-access-9rctr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:17.231517 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.231379 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e190db51-e133-4ca2-a26b-f89c2b67da45-tls-certs" pod "e190db51-e133-4ca2-a26b-f89c2b67da45" (UID: "e190db51-e133-4ca2-a26b-f89c2b67da45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:17.232934 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.232898 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-dshm" pod "e190db51-e133-4ca2-a26b-f89c2b67da45" (UID: "e190db51-e133-4ca2-a26b-f89c2b67da45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:17.329116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.329050 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:17.329116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.329095 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c98r\" (UniqueName: \"kubernetes.io/projected/e190db51-e133-4ca2-a26b-f89c2b67da45-kube-api-access-8c98r\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:17.329116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.329112 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e190db51-e133-4ca2-a26b-f89c2b67da45-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:17.329116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.329126 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:17.329432 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.329138 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:17.329432 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.329149 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd3973a3-874a-4e67-a809-60b41b853435-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:17.329432 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.329159 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:17.329432 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.329170 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rctr\" (UniqueName: \"kubernetes.io/projected/cd3973a3-874a-4e67-a809-60b41b853435-kube-api-access-9rctr\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:17.329432 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.329181 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd3973a3-874a-4e67-a809-60b41b853435-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:17.329432 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.329193 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e190db51-e133-4ca2-a26b-f89c2b67da45-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:17.461227 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.461186 4926 generic.go:356] "Generic (PLEG): container finished" podID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerID="db69411cdd265effbe3d89419b224b951d6f8e77a09f653110e2ffb61d69aa1f" exitCode=0 Oct 30 21:55:17.461707 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.461262 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" Oct 30 21:55:17.461707 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.461288 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" event={"ID":"c6f1e2cb-36a7-435d-b61e-8ab3272c591a","Type":"ContainerDied","Data":"db69411cdd265effbe3d89419b224b951d6f8e77a09f653110e2ffb61d69aa1f"} Oct 30 21:55:17.461707 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.461336 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9" event={"ID":"c6f1e2cb-36a7-435d-b61e-8ab3272c591a","Type":"ContainerDied","Data":"37b5e24f69365cf60dca91eb10c6807d5775964b1a0f9c36997aabff33e05321"} Oct 30 21:55:17.461707 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.461359 4926 scope.go:122] "RemoveContainer" containerID="db69411cdd265effbe3d89419b224b951d6f8e77a09f653110e2ffb61d69aa1f" Oct 30 21:55:17.465864 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.465835 4926 generic.go:356] "Generic (PLEG): container finished" podID="cd3973a3-874a-4e67-a809-60b41b853435" containerID="42dc88d0b40943b1564b022a0be121c14617227799ecb0751d2e296433e0d279" exitCode=0 Oct 30 21:55:17.466059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.465952 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" event={"ID":"cd3973a3-874a-4e67-a809-60b41b853435","Type":"ContainerDied","Data":"42dc88d0b40943b1564b022a0be121c14617227799ecb0751d2e296433e0d279"} Oct 30 21:55:17.466059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.465983 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" event={"ID":"cd3973a3-874a-4e67-a809-60b41b853435","Type":"ContainerDied","Data":"d3e99b87ba75ba8f4df1c762b69714e809c5ad088d5f266a7e4276883c940342"} Oct 30 21:55:17.466059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.466055 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" Oct 30 21:55:17.475164 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.475106 4926 generic.go:356] "Generic (PLEG): container finished" podID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerID="8c065bf7415ba4bd99b83fde94989ae6a27de60325cc340d5d781f6ba5aa8a59" exitCode=0 Oct 30 21:55:17.475164 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.475143 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" Oct 30 21:55:17.475338 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.475182 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" event={"ID":"e190db51-e133-4ca2-a26b-f89c2b67da45","Type":"ContainerDied","Data":"8c065bf7415ba4bd99b83fde94989ae6a27de60325cc340d5d781f6ba5aa8a59"} Oct 30 21:55:17.475338 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.475214 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh" event={"ID":"e190db51-e133-4ca2-a26b-f89c2b67da45","Type":"ContainerDied","Data":"f1ba8e2df61dba21fde115c624f6e3d43ad41d2e933132d3993ec9b4e723b330"} Oct 30 21:55:17.481521 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.481497 4926 scope.go:122] "RemoveContainer" containerID="db69411cdd265effbe3d89419b224b951d6f8e77a09f653110e2ffb61d69aa1f" Oct 30 21:55:17.481810 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:55:17.481783 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db69411cdd265effbe3d89419b224b951d6f8e77a09f653110e2ffb61d69aa1f\": container with ID starting with db69411cdd265effbe3d89419b224b951d6f8e77a09f653110e2ffb61d69aa1f not found: ID does not exist" containerID="db69411cdd265effbe3d89419b224b951d6f8e77a09f653110e2ffb61d69aa1f" Oct 30 21:55:17.481927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.481822 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db69411cdd265effbe3d89419b224b951d6f8e77a09f653110e2ffb61d69aa1f"} err="failed to get container status \"db69411cdd265effbe3d89419b224b951d6f8e77a09f653110e2ffb61d69aa1f\": rpc error: code = NotFound desc = could not find container \"db69411cdd265effbe3d89419b224b951d6f8e77a09f653110e2ffb61d69aa1f\": container with ID starting with db69411cdd265effbe3d89419b224b951d6f8e77a09f653110e2ffb61d69aa1f not found: ID does not exist" Oct 30 21:55:17.481927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.481845 4926 scope.go:122] "RemoveContainer" containerID="42dc88d0b40943b1564b022a0be121c14617227799ecb0751d2e296433e0d279" Oct 30 21:55:17.504108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.504081 4926 scope.go:122] "RemoveContainer" containerID="42dc88d0b40943b1564b022a0be121c14617227799ecb0751d2e296433e0d279" Oct 30 21:55:17.504477 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:55:17.504452 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42dc88d0b40943b1564b022a0be121c14617227799ecb0751d2e296433e0d279\": container with ID starting with 42dc88d0b40943b1564b022a0be121c14617227799ecb0751d2e296433e0d279 not found: ID does not exist" containerID="42dc88d0b40943b1564b022a0be121c14617227799ecb0751d2e296433e0d279" Oct 30 21:55:17.504531 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.504487 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42dc88d0b40943b1564b022a0be121c14617227799ecb0751d2e296433e0d279"} err="failed to get container status \"42dc88d0b40943b1564b022a0be121c14617227799ecb0751d2e296433e0d279\": rpc error: code = NotFound desc = could not find container \"42dc88d0b40943b1564b022a0be121c14617227799ecb0751d2e296433e0d279\": container with ID starting with 42dc88d0b40943b1564b022a0be121c14617227799ecb0751d2e296433e0d279 not found: ID does not exist" Oct 30 21:55:17.504531 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.504504 4926 scope.go:122] "RemoveContainer" containerID="8c065bf7415ba4bd99b83fde94989ae6a27de60325cc340d5d781f6ba5aa8a59" Oct 30 21:55:17.523215 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.523179 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9"] Oct 30 21:55:17.524590 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.524557 4926 scope.go:122] "RemoveContainer" containerID="8c065bf7415ba4bd99b83fde94989ae6a27de60325cc340d5d781f6ba5aa8a59" Oct 30 21:55:17.525082 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:55:17.525010 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c065bf7415ba4bd99b83fde94989ae6a27de60325cc340d5d781f6ba5aa8a59\": container with ID starting with 8c065bf7415ba4bd99b83fde94989ae6a27de60325cc340d5d781f6ba5aa8a59 not found: ID does not exist" containerID="8c065bf7415ba4bd99b83fde94989ae6a27de60325cc340d5d781f6ba5aa8a59" Oct 30 21:55:17.525315 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.525269 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c065bf7415ba4bd99b83fde94989ae6a27de60325cc340d5d781f6ba5aa8a59"} err="failed to get container status \"8c065bf7415ba4bd99b83fde94989ae6a27de60325cc340d5d781f6ba5aa8a59\": rpc error: code = NotFound desc = could not find container \"8c065bf7415ba4bd99b83fde94989ae6a27de60325cc340d5d781f6ba5aa8a59\": container with ID starting with 8c065bf7415ba4bd99b83fde94989ae6a27de60325cc340d5d781f6ba5aa8a59 not found: ID does not exist" Oct 30 21:55:17.527959 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.527936 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-79477e/distinct-w3-79477e-kserve-8dbd6bc4c-rrrp9"] Oct 30 21:55:17.600932 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.600899 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh"] Oct 30 21:55:17.605060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.605016 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-79477e/simulator-w3-79477e-kserve-6b747588b5-hfxrh"] Oct 30 21:55:17.630187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.630163 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s" Oct 30 21:55:17.637351 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.637326 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l"] Oct 30 21:55:17.640949 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.640927 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l"] Oct 30 21:55:17.734812 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.734722 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1cde02-c2dd-4dfb-8596-ba673bfcb5ec-kube-api-access-rqrtm\" (UniqueName: \"kubernetes.io/projected/af1cde02-c2dd-4dfb-8596-ba673bfcb5ec-kube-api-access-rqrtm\") pod \"af1cde02-c2dd-4dfb-8596-ba673bfcb5ec\" (UID: \"af1cde02-c2dd-4dfb-8596-ba673bfcb5ec\") " Oct 30 21:55:17.734812 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.734779 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af1cde02-c2dd-4dfb-8596-ba673bfcb5ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1cde02-c2dd-4dfb-8596-ba673bfcb5ec-maas-api-openshift-ca\") pod \"af1cde02-c2dd-4dfb-8596-ba673bfcb5ec\" (UID: \"af1cde02-c2dd-4dfb-8596-ba673bfcb5ec\") " Oct 30 21:55:17.735017 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.734886 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af1cde02-c2dd-4dfb-8596-ba673bfcb5ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1cde02-c2dd-4dfb-8596-ba673bfcb5ec-maas-api-xks-ca\") pod \"af1cde02-c2dd-4dfb-8596-ba673bfcb5ec\" (UID: \"af1cde02-c2dd-4dfb-8596-ba673bfcb5ec\") " Oct 30 21:55:17.735223 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.735199 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af1cde02-c2dd-4dfb-8596-ba673bfcb5ec-maas-api-openshift-ca" pod "af1cde02-c2dd-4dfb-8596-ba673bfcb5ec" (UID: "af1cde02-c2dd-4dfb-8596-ba673bfcb5ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:17.735423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.735400 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1cde02-c2dd-4dfb-8596-ba673bfcb5ec-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:17.737127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.737095 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af1cde02-c2dd-4dfb-8596-ba673bfcb5ec-maas-api-xks-ca" pod "af1cde02-c2dd-4dfb-8596-ba673bfcb5ec" (UID: "af1cde02-c2dd-4dfb-8596-ba673bfcb5ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:17.737127 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.737101 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1cde02-c2dd-4dfb-8596-ba673bfcb5ec-kube-api-access-rqrtm" pod "af1cde02-c2dd-4dfb-8596-ba673bfcb5ec" (UID: "af1cde02-c2dd-4dfb-8596-ba673bfcb5ec"). InnerVolumeSpecName "kube-api-access-rqrtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:17.837187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.837153 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqrtm\" (UniqueName: \"kubernetes.io/projected/af1cde02-c2dd-4dfb-8596-ba673bfcb5ec-kube-api-access-rqrtm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:17.837187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.837184 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1cde02-c2dd-4dfb-8596-ba673bfcb5ec-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:17.896965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:17.896924 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-79477e/premium-w3-79477e-kserve-bb884988d-6v96l" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:18.488738 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:18.488687 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s" Oct 30 21:55:18.488738 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:18.488721 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s" event={"ID":"af1cde02-c2dd-4dfb-8596-ba673bfcb5ec","Type":"ContainerDied","Data":"65eeb786593a7753572302f0af02fb04cb2c8ffd0cc0af095d71f194afe24b01"} Oct 30 21:55:18.489277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:18.488769 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65eeb786593a7753572302f0af02fb04cb2c8ffd0cc0af095d71f194afe24b01" Oct 30 21:55:18.811943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:18.811858 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:55:18.811943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:18.811913 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" podUID="7c6202f3-5168-4b2a-a167-c87f1fa98fd1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:55:18.949989 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:18.949953 4926 scope.go:122] "RemoveContainer" containerID="e5eef3cff581fa85d8fb26839d64aecf5c51ea538c0902532bc37c9078422299" Oct 30 21:55:18.962591 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:18.962568 4926 scope.go:122] "RemoveContainer" containerID="5764c27f407dd176a4e72bde98b490d4ac40e9de9b060703e81a9706bc8d5698" Oct 30 21:55:18.974423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:18.974401 4926 scope.go:122] "RemoveContainer" containerID="8bd33a8449f1e531a73babe7741f087b78cf9c1ab39fdf4daf766c4c571c300f" Oct 30 21:55:18.986688 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:18.986662 4926 scope.go:122] "RemoveContainer" containerID="8b7598fcd8b02d87bc0a34a9fc197846b5597528046fbc675e6df39774ba63cd" Oct 30 21:55:19.017626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.017598 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" path="/var/lib/kubelet/pods/c6f1e2cb-36a7-435d-b61e-8ab3272c591a/volumes" Oct 30 21:55:19.018122 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.018102 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd3973a3-874a-4e67-a809-60b41b853435" path="/var/lib/kubelet/pods/cd3973a3-874a-4e67-a809-60b41b853435/volumes" Oct 30 21:55:19.018516 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.018500 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" path="/var/lib/kubelet/pods/e190db51-e133-4ca2-a26b-f89c2b67da45/volumes" Oct 30 21:55:19.444895 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.444864 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:55:19.502556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.502490 4926 generic.go:356] "Generic (PLEG): container finished" podID="7c6202f3-5168-4b2a-a167-c87f1fa98fd1" containerID="0d914076ebb52e1a0d9e902bdead09db6e43f9b9f855dca5b8a131d1584f6026" exitCode=0 Oct 30 21:55:19.503310 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.503266 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" event={"ID":"7c6202f3-5168-4b2a-a167-c87f1fa98fd1","Type":"ContainerDied","Data":"0d914076ebb52e1a0d9e902bdead09db6e43f9b9f855dca5b8a131d1584f6026"} Oct 30 21:55:19.503444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.503342 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" event={"ID":"7c6202f3-5168-4b2a-a167-c87f1fa98fd1","Type":"ContainerDied","Data":"b76303fdd0b57e63c4b0118d84af091cc36497e3241e91dbfa0cd830f1241aa4"} Oct 30 21:55:19.503444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.503385 4926 scope.go:122] "RemoveContainer" containerID="0d914076ebb52e1a0d9e902bdead09db6e43f9b9f855dca5b8a131d1584f6026" Oct 30 21:55:19.503549 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.503531 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm" Oct 30 21:55:19.518375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.518347 4926 scope.go:122] "RemoveContainer" containerID="02609b4cdc3a2acc0da054a95129303a39cd410a1141408f1ac4fc04b737f07e" Oct 30 21:55:19.532118 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.532095 4926 scope.go:122] "RemoveContainer" containerID="0d914076ebb52e1a0d9e902bdead09db6e43f9b9f855dca5b8a131d1584f6026" Oct 30 21:55:19.532448 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:55:19.532426 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d914076ebb52e1a0d9e902bdead09db6e43f9b9f855dca5b8a131d1584f6026\": container with ID starting with 0d914076ebb52e1a0d9e902bdead09db6e43f9b9f855dca5b8a131d1584f6026 not found: ID does not exist" containerID="0d914076ebb52e1a0d9e902bdead09db6e43f9b9f855dca5b8a131d1584f6026" Oct 30 21:55:19.532534 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.532457 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d914076ebb52e1a0d9e902bdead09db6e43f9b9f855dca5b8a131d1584f6026"} err="failed to get container status \"0d914076ebb52e1a0d9e902bdead09db6e43f9b9f855dca5b8a131d1584f6026\": rpc error: code = NotFound desc = could not find container \"0d914076ebb52e1a0d9e902bdead09db6e43f9b9f855dca5b8a131d1584f6026\": container with ID starting with 0d914076ebb52e1a0d9e902bdead09db6e43f9b9f855dca5b8a131d1584f6026 not found: ID does not exist" Oct 30 21:55:19.532534 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.532476 4926 scope.go:122] "RemoveContainer" containerID="02609b4cdc3a2acc0da054a95129303a39cd410a1141408f1ac4fc04b737f07e" Oct 30 21:55:19.532817 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:55:19.532801 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02609b4cdc3a2acc0da054a95129303a39cd410a1141408f1ac4fc04b737f07e\": container with ID starting with 02609b4cdc3a2acc0da054a95129303a39cd410a1141408f1ac4fc04b737f07e not found: ID does not exist" containerID="02609b4cdc3a2acc0da054a95129303a39cd410a1141408f1ac4fc04b737f07e" Oct 30 21:55:19.532869 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.532820 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02609b4cdc3a2acc0da054a95129303a39cd410a1141408f1ac4fc04b737f07e"} err="failed to get container status \"02609b4cdc3a2acc0da054a95129303a39cd410a1141408f1ac4fc04b737f07e\": rpc error: code = NotFound desc = could not find container \"02609b4cdc3a2acc0da054a95129303a39cd410a1141408f1ac4fc04b737f07e\": container with ID starting with 02609b4cdc3a2acc0da054a95129303a39cd410a1141408f1ac4fc04b737f07e not found: ID does not exist" Oct 30 21:55:19.560809 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.560776 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istiod-ca-cert\") pod \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " Oct 30 21:55:19.560809 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.560818 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-workload-certs\") pod \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " Oct 30 21:55:19.561019 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.560869 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-token\" (UniqueName: \"kubernetes.io/projected/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-token\") pod \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " Oct 30 21:55:19.561019 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.560906 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-data\") pod \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " Oct 30 21:55:19.561019 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.560970 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-credential-socket\") pod \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " Oct 30 21:55:19.561019 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.561010 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-envoy\") pod \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " Oct 30 21:55:19.561198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.561080 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-podinfo\") pod \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " Oct 30 21:55:19.561198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.561118 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-workload-socket\") pod \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " Oct 30 21:55:19.561274 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.561197 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-kube-api-access-wxcxx\" (UniqueName: \"kubernetes.io/projected/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-kube-api-access-wxcxx\") pod \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\" (UID: \"7c6202f3-5168-4b2a-a167-c87f1fa98fd1\") " Oct 30 21:55:19.561274 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.561217 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istiod-ca-cert" pod "7c6202f3-5168-4b2a-a167-c87f1fa98fd1" (UID: "7c6202f3-5168-4b2a-a167-c87f1fa98fd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:19.561274 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.561257 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-workload-certs" pod "7c6202f3-5168-4b2a-a167-c87f1fa98fd1" (UID: "7c6202f3-5168-4b2a-a167-c87f1fa98fd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:19.561419 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.561346 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-data" pod "7c6202f3-5168-4b2a-a167-c87f1fa98fd1" (UID: "7c6202f3-5168-4b2a-a167-c87f1fa98fd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:19.561572 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.561546 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-credential-socket" pod "7c6202f3-5168-4b2a-a167-c87f1fa98fd1" (UID: "7c6202f3-5168-4b2a-a167-c87f1fa98fd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:19.561872 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.561849 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:19.562009 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.561878 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:19.562009 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.561904 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:19.562009 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.561915 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:19.562239 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.562018 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-workload-socket" pod "7c6202f3-5168-4b2a-a167-c87f1fa98fd1" (UID: "7c6202f3-5168-4b2a-a167-c87f1fa98fd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:19.563647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.563624 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-envoy" pod "7c6202f3-5168-4b2a-a167-c87f1fa98fd1" (UID: "7c6202f3-5168-4b2a-a167-c87f1fa98fd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:19.563746 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.563621 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-podinfo" pod "7c6202f3-5168-4b2a-a167-c87f1fa98fd1" (UID: "7c6202f3-5168-4b2a-a167-c87f1fa98fd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:55:19.563838 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.563783 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-token" pod "7c6202f3-5168-4b2a-a167-c87f1fa98fd1" (UID: "7c6202f3-5168-4b2a-a167-c87f1fa98fd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:19.564105 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.564086 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-kube-api-access-wxcxx" pod "7c6202f3-5168-4b2a-a167-c87f1fa98fd1" (UID: "7c6202f3-5168-4b2a-a167-c87f1fa98fd1"). InnerVolumeSpecName "kube-api-access-wxcxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:19.663608 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.663569 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:19.663608 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.663606 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:19.663608 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.663616 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:19.663867 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.663626 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxcxx\" (UniqueName: \"kubernetes.io/projected/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-kube-api-access-wxcxx\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:19.663867 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.663637 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c6202f3-5168-4b2a-a167-c87f1fa98fd1-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:19.859168 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.859064 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm"] Oct 30 21:55:19.863327 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:19.863290 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-66cc54-openshift-default-745899c56f-sgmcm"] Oct 30 21:55:20.031877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:20.031716 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7"] Oct 30 21:55:20.036404 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:20.036373 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v"] Oct 30 21:55:20.040002 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:20.039976 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ccec6d1-gfql7"] Oct 30 21:55:20.043622 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:20.043596 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-161e2a60-78b7v"] Oct 30 21:55:21.016822 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:21.016786 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c6202f3-5168-4b2a-a167-c87f1fa98fd1" path="/var/lib/kubelet/pods/7c6202f3-5168-4b2a-a167-c87f1fa98fd1/volumes" Oct 30 21:55:21.017509 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:21.017485 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3920bbc-2a92-438d-ab8a-d4e076d3602d" path="/var/lib/kubelet/pods/b3920bbc-2a92-438d-ab8a-d4e076d3602d/volumes" Oct 30 21:55:21.017978 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:21.017958 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d83901a8-dbdf-4385-8e0f-af7d2fa13ccf" path="/var/lib/kubelet/pods/d83901a8-dbdf-4385-8e0f-af7d2fa13ccf/volumes" Oct 30 21:55:21.920148 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:21.920110 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j"] Oct 30 21:55:21.920372 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:21.920311 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" podUID="c58ef83e-f70c-499e-8377-63d4e375601b" containerName="istio-proxy" containerID="cri-o://d70e82987d246b50bf5b4d97b4c3dfe5a2180da88e2126101024c4c63250570c" gracePeriod=30 Oct 30 21:55:22.503008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.502968 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j"] Oct 30 21:55:22.503521 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.503507 4926 cpu_manager.go:414] "containerMap: removing container" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" Oct 30 21:55:22.503564 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.503521 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" Oct 30 21:55:22.503564 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.503539 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" Oct 30 21:55:22.503564 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.503543 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" Oct 30 21:55:22.503564 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.503557 4926 cpu_manager.go:414] "containerMap: removing container" podUID="7c6202f3-5168-4b2a-a167-c87f1fa98fd1" containerName="istio-proxy" Oct 30 21:55:22.503564 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.503562 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c6202f3-5168-4b2a-a167-c87f1fa98fd1" containerName="istio-proxy" Oct 30 21:55:22.503720 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.503582 4926 cpu_manager.go:414] "containerMap: removing container" podUID="50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" containerName="istio-proxy" Oct 30 21:55:22.503720 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.503587 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" containerName="istio-proxy" Oct 30 21:55:22.503720 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.503595 4926 cpu_manager.go:414] "containerMap: removing container" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" Oct 30 21:55:22.503720 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.503599 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" Oct 30 21:55:22.503720 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.503607 4926 cpu_manager.go:414] "containerMap: removing container" podUID="af1cde02-c2dd-4dfb-8596-ba673bfcb5ec" containerName="revoke-keys" Oct 30 21:55:22.503720 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.503611 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1cde02-c2dd-4dfb-8596-ba673bfcb5ec" containerName="revoke-keys" Oct 30 21:55:22.503720 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.503626 4926 cpu_manager.go:414] "containerMap: removing container" podUID="50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" containerName="istio-proxy" Oct 30 21:55:22.503720 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.503631 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" containerName="istio-proxy" Oct 30 21:55:22.503720 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.503709 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1cde02-c2dd-4dfb-8596-ba673bfcb5ec" containerName="revoke-keys" Oct 30 21:55:22.503720 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.503722 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" containerName="istio-proxy" Oct 30 21:55:22.504025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.503728 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c6202f3-5168-4b2a-a167-c87f1fa98fd1" containerName="istio-proxy" Oct 30 21:55:22.504025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.503735 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="e190db51-e133-4ca2-a26b-f89c2b67da45" containerName="main" Oct 30 21:55:22.504025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.503742 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f1e2cb-36a7-435d-b61e-8ab3272c591a" containerName="main" Oct 30 21:55:22.504025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.503752 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="50bd09f8-bbc4-41f3-ae8f-f517e50a78b5" containerName="istio-proxy" Oct 30 21:55:22.504025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.503758 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd3973a3-874a-4e67-a809-60b41b853435" containerName="main" Oct 30 21:55:22.509116 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.509095 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j" Oct 30 21:55:22.530328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.530286 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j"] Oct 30 21:55:22.536079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.535927 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:55:22.536079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.535988 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:55:22.601493 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.601448 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbp5p\" (UniqueName: \"kubernetes.io/projected/88d0b36e-2321-4b48-a80b-9428ef221151-kube-api-access-lbp5p\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j\" (UID: \"88d0b36e-2321-4b48-a80b-9428ef221151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j" Oct 30 21:55:22.601721 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.601679 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d0b36e-2321-4b48-a80b-9428ef221151-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j\" (UID: \"88d0b36e-2321-4b48-a80b-9428ef221151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j" Oct 30 21:55:22.601864 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.601847 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d0b36e-2321-4b48-a80b-9428ef221151-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j\" (UID: \"88d0b36e-2321-4b48-a80b-9428ef221151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j" Oct 30 21:55:22.703075 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.703017 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d0b36e-2321-4b48-a80b-9428ef221151-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j\" (UID: \"88d0b36e-2321-4b48-a80b-9428ef221151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j" Oct 30 21:55:22.703268 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.703171 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d0b36e-2321-4b48-a80b-9428ef221151-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j\" (UID: \"88d0b36e-2321-4b48-a80b-9428ef221151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j" Oct 30 21:55:22.703354 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.703269 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbp5p\" (UniqueName: \"kubernetes.io/projected/88d0b36e-2321-4b48-a80b-9428ef221151-kube-api-access-lbp5p\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j\" (UID: \"88d0b36e-2321-4b48-a80b-9428ef221151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j" Oct 30 21:55:22.704008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.703975 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d0b36e-2321-4b48-a80b-9428ef221151-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j\" (UID: \"88d0b36e-2321-4b48-a80b-9428ef221151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j" Oct 30 21:55:22.705649 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.705618 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d0b36e-2321-4b48-a80b-9428ef221151-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j\" (UID: \"88d0b36e-2321-4b48-a80b-9428ef221151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j" Oct 30 21:55:22.722971 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.722940 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbp5p\" (UniqueName: \"kubernetes.io/projected/88d0b36e-2321-4b48-a80b-9428ef221151-kube-api-access-lbp5p\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j\" (UID: \"88d0b36e-2321-4b48-a80b-9428ef221151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j" Oct 30 21:55:22.823175 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:22.823095 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j" Oct 30 21:55:23.141700 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:23.141667 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j"] Oct 30 21:55:23.141850 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:55:23.141766 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d0b36e_2321_4b48_a80b_9428ef221151.slice/crio-3ef0491cc20f91378962ab3f47f6a6483c63a9f95e4dfd49299e5d1cd717d66c WatchSource:0}: Error finding container 3ef0491cc20f91378962ab3f47f6a6483c63a9f95e4dfd49299e5d1cd717d66c: Status 404 returned error can't find the container with id 3ef0491cc20f91378962ab3f47f6a6483c63a9f95e4dfd49299e5d1cd717d66c Oct 30 21:55:23.546109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:23.546064 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j" event={"ID":"88d0b36e-2321-4b48-a80b-9428ef221151","Type":"ContainerStarted","Data":"3ef0491cc20f91378962ab3f47f6a6483c63a9f95e4dfd49299e5d1cd717d66c"} Oct 30 21:55:24.564322 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:24.564286 4926 generic.go:356] "Generic (PLEG): container finished" podID="88d0b36e-2321-4b48-a80b-9428ef221151" containerID="338db3482bf099d6b1eb6df1a06a97edf8acc7806e40ab6915aea9a005c14319" exitCode=0 Oct 30 21:55:24.564866 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:24.564377 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j" event={"ID":"88d0b36e-2321-4b48-a80b-9428ef221151","Type":"ContainerDied","Data":"338db3482bf099d6b1eb6df1a06a97edf8acc7806e40ab6915aea9a005c14319"} Oct 30 21:55:25.828454 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:25.828303 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j" Oct 30 21:55:25.846313 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:25.846281 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88d0b36e-2321-4b48-a80b-9428ef221151-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d0b36e-2321-4b48-a80b-9428ef221151-maas-api-openshift-ca\") pod \"88d0b36e-2321-4b48-a80b-9428ef221151\" (UID: \"88d0b36e-2321-4b48-a80b-9428ef221151\") " Oct 30 21:55:25.846488 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:25.846457 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88d0b36e-2321-4b48-a80b-9428ef221151-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d0b36e-2321-4b48-a80b-9428ef221151-maas-api-xks-ca\") pod \"88d0b36e-2321-4b48-a80b-9428ef221151\" (UID: \"88d0b36e-2321-4b48-a80b-9428ef221151\") " Oct 30 21:55:25.846552 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:25.846533 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88d0b36e-2321-4b48-a80b-9428ef221151-kube-api-access-lbp5p\" (UniqueName: \"kubernetes.io/projected/88d0b36e-2321-4b48-a80b-9428ef221151-kube-api-access-lbp5p\") pod \"88d0b36e-2321-4b48-a80b-9428ef221151\" (UID: \"88d0b36e-2321-4b48-a80b-9428ef221151\") " Oct 30 21:55:25.846711 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:25.846689 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d0b36e-2321-4b48-a80b-9428ef221151-maas-api-openshift-ca" pod "88d0b36e-2321-4b48-a80b-9428ef221151" (UID: "88d0b36e-2321-4b48-a80b-9428ef221151"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:25.847004 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:25.846986 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d0b36e-2321-4b48-a80b-9428ef221151-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:25.848694 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:25.848669 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d0b36e-2321-4b48-a80b-9428ef221151-kube-api-access-lbp5p" pod "88d0b36e-2321-4b48-a80b-9428ef221151" (UID: "88d0b36e-2321-4b48-a80b-9428ef221151"). InnerVolumeSpecName "kube-api-access-lbp5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:25.848784 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:25.848695 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d0b36e-2321-4b48-a80b-9428ef221151-maas-api-xks-ca" pod "88d0b36e-2321-4b48-a80b-9428ef221151" (UID: "88d0b36e-2321-4b48-a80b-9428ef221151"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:25.948246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:25.948210 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d0b36e-2321-4b48-a80b-9428ef221151-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:25.948246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:25.948244 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbp5p\" (UniqueName: \"kubernetes.io/projected/88d0b36e-2321-4b48-a80b-9428ef221151-kube-api-access-lbp5p\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:26.585005 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:26.584954 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j" event={"ID":"88d0b36e-2321-4b48-a80b-9428ef221151","Type":"ContainerDied","Data":"3ef0491cc20f91378962ab3f47f6a6483c63a9f95e4dfd49299e5d1cd717d66c"} Oct 30 21:55:26.585232 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:26.584970 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j" Oct 30 21:55:26.585232 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:26.585066 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ef0491cc20f91378962ab3f47f6a6483c63a9f95e4dfd49299e5d1cd717d66c" Oct 30 21:55:27.210232 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.210204 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:55:27.263647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.263611 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c58ef83e-f70c-499e-8377-63d4e375601b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58ef83e-f70c-499e-8377-63d4e375601b-istiod-ca-cert\") pod \"c58ef83e-f70c-499e-8377-63d4e375601b\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " Oct 30 21:55:27.263877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.263731 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c58ef83e-f70c-499e-8377-63d4e375601b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58ef83e-f70c-499e-8377-63d4e375601b-istio-podinfo\") pod \"c58ef83e-f70c-499e-8377-63d4e375601b\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " Oct 30 21:55:27.264155 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.264104 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c58ef83e-f70c-499e-8377-63d4e375601b-istiod-ca-cert" pod "c58ef83e-f70c-499e-8377-63d4e375601b" (UID: "c58ef83e-f70c-499e-8377-63d4e375601b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:27.264703 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.264200 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-istio-envoy\") pod \"c58ef83e-f70c-499e-8377-63d4e375601b\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " Oct 30 21:55:27.264816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.264732 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58ef83e-f70c-499e-8377-63d4e375601b-kube-api-access-hv5k4\" (UniqueName: \"kubernetes.io/projected/c58ef83e-f70c-499e-8377-63d4e375601b-kube-api-access-hv5k4\") pod \"c58ef83e-f70c-499e-8377-63d4e375601b\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " Oct 30 21:55:27.264816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.264766 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-workload-socket\") pod \"c58ef83e-f70c-499e-8377-63d4e375601b\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " Oct 30 21:55:27.264816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.264787 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c58ef83e-f70c-499e-8377-63d4e375601b-istio-token\" (UniqueName: \"kubernetes.io/projected/c58ef83e-f70c-499e-8377-63d4e375601b-istio-token\") pod \"c58ef83e-f70c-499e-8377-63d4e375601b\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " Oct 30 21:55:27.265005 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.264846 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-workload-certs\") pod \"c58ef83e-f70c-499e-8377-63d4e375601b\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " Oct 30 21:55:27.265005 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.264902 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-istio-data\") pod \"c58ef83e-f70c-499e-8377-63d4e375601b\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " Oct 30 21:55:27.265776 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.265749 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-credential-socket\") pod \"c58ef83e-f70c-499e-8377-63d4e375601b\" (UID: \"c58ef83e-f70c-499e-8377-63d4e375601b\") " Oct 30 21:55:27.279079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.266964 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-workload-socket" pod "c58ef83e-f70c-499e-8377-63d4e375601b" (UID: "c58ef83e-f70c-499e-8377-63d4e375601b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.279079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.267237 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-workload-certs" pod "c58ef83e-f70c-499e-8377-63d4e375601b" (UID: "c58ef83e-f70c-499e-8377-63d4e375601b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.279079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.267436 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-istio-data" pod "c58ef83e-f70c-499e-8377-63d4e375601b" (UID: "c58ef83e-f70c-499e-8377-63d4e375601b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.279079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.267705 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-credential-socket" pod "c58ef83e-f70c-499e-8377-63d4e375601b" (UID: "c58ef83e-f70c-499e-8377-63d4e375601b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.279079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.267738 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-istio-envoy" pod "c58ef83e-f70c-499e-8377-63d4e375601b" (UID: "c58ef83e-f70c-499e-8377-63d4e375601b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:27.279079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.271077 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58ef83e-f70c-499e-8377-63d4e375601b-kube-api-access-hv5k4" pod "c58ef83e-f70c-499e-8377-63d4e375601b" (UID: "c58ef83e-f70c-499e-8377-63d4e375601b"). InnerVolumeSpecName "kube-api-access-hv5k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:27.279079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.271112 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c58ef83e-f70c-499e-8377-63d4e375601b-istio-token" pod "c58ef83e-f70c-499e-8377-63d4e375601b" (UID: "c58ef83e-f70c-499e-8377-63d4e375601b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:27.279079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.272417 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c58ef83e-f70c-499e-8377-63d4e375601b-istio-podinfo" pod "c58ef83e-f70c-499e-8377-63d4e375601b" (UID: "c58ef83e-f70c-499e-8377-63d4e375601b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:55:27.280151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.280128 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c58ef83e-f70c-499e-8377-63d4e375601b-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:27.280228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.280157 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:27.280228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.280172 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv5k4\" (UniqueName: \"kubernetes.io/projected/c58ef83e-f70c-499e-8377-63d4e375601b-kube-api-access-hv5k4\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:27.280228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.280188 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:27.280228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.280200 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c58ef83e-f70c-499e-8377-63d4e375601b-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:27.280228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.280211 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:27.280228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.280224 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:27.280447 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.280237 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c58ef83e-f70c-499e-8377-63d4e375601b-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:27.280447 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.280248 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c58ef83e-f70c-499e-8377-63d4e375601b-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:27.599300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.599206 4926 generic.go:356] "Generic (PLEG): container finished" podID="c58ef83e-f70c-499e-8377-63d4e375601b" containerID="d70e82987d246b50bf5b4d97b4c3dfe5a2180da88e2126101024c4c63250570c" exitCode=0 Oct 30 21:55:27.599300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.599271 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" Oct 30 21:55:27.599491 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.599289 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" event={"ID":"c58ef83e-f70c-499e-8377-63d4e375601b","Type":"ContainerDied","Data":"d70e82987d246b50bf5b4d97b4c3dfe5a2180da88e2126101024c4c63250570c"} Oct 30 21:55:27.599491 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.599370 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j" event={"ID":"c58ef83e-f70c-499e-8377-63d4e375601b","Type":"ContainerDied","Data":"378e929b12b3fc8288c1d0634af0d169ebde36890945ce8796baa8666d6b080b"} Oct 30 21:55:27.599491 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.599403 4926 scope.go:122] "RemoveContainer" containerID="d70e82987d246b50bf5b4d97b4c3dfe5a2180da88e2126101024c4c63250570c" Oct 30 21:55:27.613736 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.613709 4926 scope.go:122] "RemoveContainer" containerID="3c7125df7f2c6e991d35ab60f6ea98d175725029b42cae56451c9824a3de73e4" Oct 30 21:55:27.627195 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.627114 4926 scope.go:122] "RemoveContainer" containerID="d70e82987d246b50bf5b4d97b4c3dfe5a2180da88e2126101024c4c63250570c" Oct 30 21:55:27.627459 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:55:27.627432 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d70e82987d246b50bf5b4d97b4c3dfe5a2180da88e2126101024c4c63250570c\": container with ID starting with d70e82987d246b50bf5b4d97b4c3dfe5a2180da88e2126101024c4c63250570c not found: ID does not exist" containerID="d70e82987d246b50bf5b4d97b4c3dfe5a2180da88e2126101024c4c63250570c" Oct 30 21:55:27.627535 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.627467 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70e82987d246b50bf5b4d97b4c3dfe5a2180da88e2126101024c4c63250570c"} err="failed to get container status \"d70e82987d246b50bf5b4d97b4c3dfe5a2180da88e2126101024c4c63250570c\": rpc error: code = NotFound desc = could not find container \"d70e82987d246b50bf5b4d97b4c3dfe5a2180da88e2126101024c4c63250570c\": container with ID starting with d70e82987d246b50bf5b4d97b4c3dfe5a2180da88e2126101024c4c63250570c not found: ID does not exist" Oct 30 21:55:27.627535 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.627485 4926 scope.go:122] "RemoveContainer" containerID="3c7125df7f2c6e991d35ab60f6ea98d175725029b42cae56451c9824a3de73e4" Oct 30 21:55:27.627737 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:55:27.627715 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c7125df7f2c6e991d35ab60f6ea98d175725029b42cae56451c9824a3de73e4\": container with ID starting with 3c7125df7f2c6e991d35ab60f6ea98d175725029b42cae56451c9824a3de73e4 not found: ID does not exist" containerID="3c7125df7f2c6e991d35ab60f6ea98d175725029b42cae56451c9824a3de73e4" Oct 30 21:55:27.627793 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.627744 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c7125df7f2c6e991d35ab60f6ea98d175725029b42cae56451c9824a3de73e4"} err="failed to get container status \"3c7125df7f2c6e991d35ab60f6ea98d175725029b42cae56451c9824a3de73e4\": rpc error: code = NotFound desc = could not find container \"3c7125df7f2c6e991d35ab60f6ea98d175725029b42cae56451c9824a3de73e4\": container with ID starting with 3c7125df7f2c6e991d35ab60f6ea98d175725029b42cae56451c9824a3de73e4 not found: ID does not exist" Oct 30 21:55:27.651936 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.651904 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j"] Oct 30 21:55:27.662748 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:27.662719 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-f5bef0-openshift-default-f946cb9b5-zvm8j"] Oct 30 21:55:28.406928 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.406891 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9"] Oct 30 21:55:28.407469 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.407160 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" podUID="963cb97c-107b-440f-8927-10dda8d1628b" containerName="maas-api" containerID="cri-o://5e743a0b45f6970cfd906055353fcea5908888f1808a432e6a0fa61944e57e40" gracePeriod=30 Oct 30 21:55:28.628358 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.628307 4926 generic.go:356] "Generic (PLEG): container finished" podID="963cb97c-107b-440f-8927-10dda8d1628b" containerID="5e743a0b45f6970cfd906055353fcea5908888f1808a432e6a0fa61944e57e40" exitCode=0 Oct 30 21:55:28.628544 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.628445 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" event={"ID":"963cb97c-107b-440f-8927-10dda8d1628b","Type":"ContainerDied","Data":"5e743a0b45f6970cfd906055353fcea5908888f1808a432e6a0fa61944e57e40"} Oct 30 21:55:28.761735 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.761708 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" Oct 30 21:55:28.800995 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.800950 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963cb97c-107b-440f-8927-10dda8d1628b-kube-api-access-6dwp9\" (UniqueName: \"kubernetes.io/projected/963cb97c-107b-440f-8927-10dda8d1628b-kube-api-access-6dwp9\") pod \"963cb97c-107b-440f-8927-10dda8d1628b\" (UID: \"963cb97c-107b-440f-8927-10dda8d1628b\") " Oct 30 21:55:28.801213 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.801127 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/963cb97c-107b-440f-8927-10dda8d1628b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963cb97c-107b-440f-8927-10dda8d1628b-maas-api-tls\") pod \"963cb97c-107b-440f-8927-10dda8d1628b\" (UID: \"963cb97c-107b-440f-8927-10dda8d1628b\") " Oct 30 21:55:28.801364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.801215 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/963cb97c-107b-440f-8927-10dda8d1628b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/963cb97c-107b-440f-8927-10dda8d1628b-metrics-certs\") pod \"963cb97c-107b-440f-8927-10dda8d1628b\" (UID: \"963cb97c-107b-440f-8927-10dda8d1628b\") " Oct 30 21:55:28.801364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.801297 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/963cb97c-107b-440f-8927-10dda8d1628b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963cb97c-107b-440f-8927-10dda8d1628b-openshift-service-ca\") pod \"963cb97c-107b-440f-8927-10dda8d1628b\" (UID: \"963cb97c-107b-440f-8927-10dda8d1628b\") " Oct 30 21:55:28.802077 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.802014 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963cb97c-107b-440f-8927-10dda8d1628b-openshift-service-ca" pod "963cb97c-107b-440f-8927-10dda8d1628b" (UID: "963cb97c-107b-440f-8927-10dda8d1628b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:28.803513 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.803483 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963cb97c-107b-440f-8927-10dda8d1628b-kube-api-access-6dwp9" pod "963cb97c-107b-440f-8927-10dda8d1628b" (UID: "963cb97c-107b-440f-8927-10dda8d1628b"). InnerVolumeSpecName "kube-api-access-6dwp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:28.803796 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.803768 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963cb97c-107b-440f-8927-10dda8d1628b-maas-api-tls" pod "963cb97c-107b-440f-8927-10dda8d1628b" (UID: "963cb97c-107b-440f-8927-10dda8d1628b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:28.803796 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.803784 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/963cb97c-107b-440f-8927-10dda8d1628b-metrics-certs" pod "963cb97c-107b-440f-8927-10dda8d1628b" (UID: "963cb97c-107b-440f-8927-10dda8d1628b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:28.953065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.945017 4926 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/963cb97c-107b-440f-8927-10dda8d1628b-openshift-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:28.953065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.945083 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dwp9\" (UniqueName: \"kubernetes.io/projected/963cb97c-107b-440f-8927-10dda8d1628b-kube-api-access-6dwp9\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:28.953065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.945099 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/963cb97c-107b-440f-8927-10dda8d1628b-maas-api-tls\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:28.953065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.945113 4926 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/963cb97c-107b-440f-8927-10dda8d1628b-metrics-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:28.968395 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.967935 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6"] Oct 30 21:55:28.968838 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.968794 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c58ef83e-f70c-499e-8377-63d4e375601b" containerName="istio-proxy" Oct 30 21:55:28.968838 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.968825 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c58ef83e-f70c-499e-8377-63d4e375601b" containerName="istio-proxy" Oct 30 21:55:28.969193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.968859 4926 cpu_manager.go:414] "containerMap: removing container" podUID="7c6202f3-5168-4b2a-a167-c87f1fa98fd1" containerName="istio-proxy" Oct 30 21:55:28.969193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.968868 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c6202f3-5168-4b2a-a167-c87f1fa98fd1" containerName="istio-proxy" Oct 30 21:55:28.969193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.968916 4926 cpu_manager.go:414] "containerMap: removing container" podUID="88d0b36e-2321-4b48-a80b-9428ef221151" containerName="revoke-keys" Oct 30 21:55:28.969193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.968924 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88d0b36e-2321-4b48-a80b-9428ef221151" containerName="revoke-keys" Oct 30 21:55:28.969193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.968935 4926 cpu_manager.go:414] "containerMap: removing container" podUID="963cb97c-107b-440f-8927-10dda8d1628b" containerName="maas-api" Oct 30 21:55:28.969193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.968941 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963cb97c-107b-440f-8927-10dda8d1628b" containerName="maas-api" Oct 30 21:55:28.969193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.969110 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c58ef83e-f70c-499e-8377-63d4e375601b" containerName="istio-proxy" Oct 30 21:55:28.969193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.969128 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c58ef83e-f70c-499e-8377-63d4e375601b" containerName="istio-proxy" Oct 30 21:55:28.969193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.969141 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="88d0b36e-2321-4b48-a80b-9428ef221151" containerName="revoke-keys" Oct 30 21:55:28.969193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.969151 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="963cb97c-107b-440f-8927-10dda8d1628b" containerName="maas-api" Oct 30 21:55:28.969193 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.969161 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c6202f3-5168-4b2a-a167-c87f1fa98fd1" containerName="istio-proxy" Oct 30 21:55:28.996787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:28.994202 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6" Oct 30 21:55:29.009871 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.009819 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de26f85c-bkrd7"] Oct 30 21:55:29.010914 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.010882 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c58ef83e-f70c-499e-8377-63d4e375601b" containerName="istio-proxy" Oct 30 21:55:29.011704 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.011681 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c58ef83e-f70c-499e-8377-63d4e375601b" containerName="istio-proxy" Oct 30 21:55:29.011931 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.011607 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:55:29.012017 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.011656 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:55:29.023159 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.023131 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de26f85c-bkrd7" Oct 30 21:55:29.034917 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.034890 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c58ef83e-f70c-499e-8377-63d4e375601b" path="/var/lib/kubelet/pods/c58ef83e-f70c-499e-8377-63d4e375601b/volumes" Oct 30 21:55:29.035545 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.035522 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6"] Oct 30 21:55:29.035646 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.035570 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de26f85c-bkrd7"] Oct 30 21:55:29.047122 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.047097 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c19f7b-d721-4ffa-9b72-a671ea117191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dea2ab3-mpmm6\" (UID: \"d0c19f7b-d721-4ffa-9b72-a671ea117191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6" Oct 30 21:55:29.047535 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.047517 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kdwx\" (UniqueName: \"kubernetes.io/projected/d0c19f7b-d721-4ffa-9b72-a671ea117191-kube-api-access-5kdwx\") pod \"maas-api-revoke-sub-5dea2ab3-mpmm6\" (UID: \"d0c19f7b-d721-4ffa-9b72-a671ea117191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6" Oct 30 21:55:29.047939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.047924 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c19f7b-d721-4ffa-9b72-a671ea117191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dea2ab3-mpmm6\" (UID: \"d0c19f7b-d721-4ffa-9b72-a671ea117191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6" Oct 30 21:55:29.148647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.148603 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rntrn\" (UniqueName: \"kubernetes.io/projected/4931c025-ff04-44cf-80ea-88c1d97df1e9-kube-api-access-rntrn\") pod \"maas-api-revoke-sub-de26f85c-bkrd7\" (UID: \"4931c025-ff04-44cf-80ea-88c1d97df1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de26f85c-bkrd7" Oct 30 21:55:29.148835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.148737 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c19f7b-d721-4ffa-9b72-a671ea117191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dea2ab3-mpmm6\" (UID: \"d0c19f7b-d721-4ffa-9b72-a671ea117191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6" Oct 30 21:55:29.148835 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.148805 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kdwx\" (UniqueName: \"kubernetes.io/projected/d0c19f7b-d721-4ffa-9b72-a671ea117191-kube-api-access-5kdwx\") pod \"maas-api-revoke-sub-5dea2ab3-mpmm6\" (UID: \"d0c19f7b-d721-4ffa-9b72-a671ea117191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6" Oct 30 21:55:29.148904 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.148863 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4931c025-ff04-44cf-80ea-88c1d97df1e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de26f85c-bkrd7\" (UID: \"4931c025-ff04-44cf-80ea-88c1d97df1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de26f85c-bkrd7" Oct 30 21:55:29.148904 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.148882 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c19f7b-d721-4ffa-9b72-a671ea117191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dea2ab3-mpmm6\" (UID: \"d0c19f7b-d721-4ffa-9b72-a671ea117191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6" Oct 30 21:55:29.149088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.149065 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4931c025-ff04-44cf-80ea-88c1d97df1e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de26f85c-bkrd7\" (UID: \"4931c025-ff04-44cf-80ea-88c1d97df1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de26f85c-bkrd7" Oct 30 21:55:29.149516 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.149494 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c19f7b-d721-4ffa-9b72-a671ea117191-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5dea2ab3-mpmm6\" (UID: \"d0c19f7b-d721-4ffa-9b72-a671ea117191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6" Oct 30 21:55:29.151370 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.151345 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c19f7b-d721-4ffa-9b72-a671ea117191-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5dea2ab3-mpmm6\" (UID: \"d0c19f7b-d721-4ffa-9b72-a671ea117191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6" Oct 30 21:55:29.169129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.169096 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kdwx\" (UniqueName: \"kubernetes.io/projected/d0c19f7b-d721-4ffa-9b72-a671ea117191-kube-api-access-5kdwx\") pod \"maas-api-revoke-sub-5dea2ab3-mpmm6\" (UID: \"d0c19f7b-d721-4ffa-9b72-a671ea117191\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6" Oct 30 21:55:29.250345 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.250263 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4931c025-ff04-44cf-80ea-88c1d97df1e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de26f85c-bkrd7\" (UID: \"4931c025-ff04-44cf-80ea-88c1d97df1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de26f85c-bkrd7" Oct 30 21:55:29.250529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.250345 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rntrn\" (UniqueName: \"kubernetes.io/projected/4931c025-ff04-44cf-80ea-88c1d97df1e9-kube-api-access-rntrn\") pod \"maas-api-revoke-sub-de26f85c-bkrd7\" (UID: \"4931c025-ff04-44cf-80ea-88c1d97df1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de26f85c-bkrd7" Oct 30 21:55:29.250529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.250490 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4931c025-ff04-44cf-80ea-88c1d97df1e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de26f85c-bkrd7\" (UID: \"4931c025-ff04-44cf-80ea-88c1d97df1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de26f85c-bkrd7" Oct 30 21:55:29.250970 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.250952 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4931c025-ff04-44cf-80ea-88c1d97df1e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de26f85c-bkrd7\" (UID: \"4931c025-ff04-44cf-80ea-88c1d97df1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de26f85c-bkrd7" Oct 30 21:55:29.253103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.253078 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4931c025-ff04-44cf-80ea-88c1d97df1e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de26f85c-bkrd7\" (UID: \"4931c025-ff04-44cf-80ea-88c1d97df1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de26f85c-bkrd7" Oct 30 21:55:29.276432 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.276405 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rntrn\" (UniqueName: \"kubernetes.io/projected/4931c025-ff04-44cf-80ea-88c1d97df1e9-kube-api-access-rntrn\") pod \"maas-api-revoke-sub-de26f85c-bkrd7\" (UID: \"4931c025-ff04-44cf-80ea-88c1d97df1e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de26f85c-bkrd7" Oct 30 21:55:29.322311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.322275 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b"] Oct 30 21:55:29.329624 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.329593 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6" Oct 30 21:55:29.330605 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.330576 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-22d912-jbs8b"] Oct 30 21:55:29.343474 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.343431 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de26f85c-bkrd7" Oct 30 21:55:29.638460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.638381 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" event={"ID":"963cb97c-107b-440f-8927-10dda8d1628b","Type":"ContainerDied","Data":"ed9f2432f1b397e803aaa82d744285f54a35f2896ab984036bc0d46598ffef65"} Oct 30 21:55:29.638460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.638403 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9" Oct 30 21:55:29.638460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.638442 4926 scope.go:122] "RemoveContainer" containerID="5e743a0b45f6970cfd906055353fcea5908888f1808a432e6a0fa61944e57e40" Oct 30 21:55:29.657790 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:55:29.657753 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c19f7b_d721_4ffa_9b72_a671ea117191.slice/crio-cb88d4f09400e3b5c5135699644119215e2e916332d81e1510b43a368ca9c71d WatchSource:0}: Error finding container cb88d4f09400e3b5c5135699644119215e2e916332d81e1510b43a368ca9c71d: Status 404 returned error can't find the container with id cb88d4f09400e3b5c5135699644119215e2e916332d81e1510b43a368ca9c71d Oct 30 21:55:29.658842 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.658812 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6"] Oct 30 21:55:29.698055 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.677390 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de26f85c-bkrd7"] Oct 30 21:55:29.759648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.754610 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9"] Oct 30 21:55:29.759648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:29.754666 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-22d912-84c999b7bd-8t8g9"] Oct 30 21:55:30.648810 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:30.648772 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6" event={"ID":"d0c19f7b-d721-4ffa-9b72-a671ea117191","Type":"ContainerStarted","Data":"8b06432ae85232e930c9d64301210f515fde2dc9b467a32a7ee1ab3a2abc8447"} Oct 30 21:55:30.649300 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:30.648830 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6" event={"ID":"d0c19f7b-d721-4ffa-9b72-a671ea117191","Type":"ContainerStarted","Data":"cb88d4f09400e3b5c5135699644119215e2e916332d81e1510b43a368ca9c71d"} Oct 30 21:55:30.651501 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:30.651475 4926 generic.go:356] "Generic (PLEG): container finished" podID="4931c025-ff04-44cf-80ea-88c1d97df1e9" containerID="f4023f8d251f50ec42a8d88464fa9ef8f15478d2328e0cff3892e5290ab24c5a" exitCode=0 Oct 30 21:55:30.651613 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:30.651510 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de26f85c-bkrd7" event={"ID":"4931c025-ff04-44cf-80ea-88c1d97df1e9","Type":"ContainerDied","Data":"f4023f8d251f50ec42a8d88464fa9ef8f15478d2328e0cff3892e5290ab24c5a"} Oct 30 21:55:30.651613 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:30.651540 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de26f85c-bkrd7" event={"ID":"4931c025-ff04-44cf-80ea-88c1d97df1e9","Type":"ContainerStarted","Data":"120771f01ba6913f18b98f7194f1c2ddcccf38182d24c7da1a09f3e6d9babd32"} Oct 30 21:55:30.691662 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:30.691605 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6" podStartSLOduration=2.691590622 podStartE2EDuration="2.691590622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:30.687923615 +0000 UTC m=+3317.808997360" watchObservedRunningTime="2026-10-30 21:55:30.691590622 +0000 UTC m=+3317.812664403" Oct 30 21:55:31.018806 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:31.018769 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67c5b659-8710-4d3f-8bc9-66b6579b26a7" path="/var/lib/kubelet/pods/67c5b659-8710-4d3f-8bc9-66b6579b26a7/volumes" Oct 30 21:55:31.019303 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:31.019288 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="963cb97c-107b-440f-8927-10dda8d1628b" path="/var/lib/kubelet/pods/963cb97c-107b-440f-8927-10dda8d1628b/volumes" Oct 30 21:55:31.662549 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:31.662509 4926 generic.go:356] "Generic (PLEG): container finished" podID="d0c19f7b-d721-4ffa-9b72-a671ea117191" containerID="8b06432ae85232e930c9d64301210f515fde2dc9b467a32a7ee1ab3a2abc8447" exitCode=0 Oct 30 21:55:31.662959 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:31.662592 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6" event={"ID":"d0c19f7b-d721-4ffa-9b72-a671ea117191","Type":"ContainerDied","Data":"8b06432ae85232e930c9d64301210f515fde2dc9b467a32a7ee1ab3a2abc8447"} Oct 30 21:55:31.840986 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:31.840960 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de26f85c-bkrd7" Oct 30 21:55:31.890562 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:31.890522 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4931c025-ff04-44cf-80ea-88c1d97df1e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4931c025-ff04-44cf-80ea-88c1d97df1e9-maas-api-xks-ca\") pod \"4931c025-ff04-44cf-80ea-88c1d97df1e9\" (UID: \"4931c025-ff04-44cf-80ea-88c1d97df1e9\") " Oct 30 21:55:31.890745 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:31.890660 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4931c025-ff04-44cf-80ea-88c1d97df1e9-kube-api-access-rntrn\" (UniqueName: \"kubernetes.io/projected/4931c025-ff04-44cf-80ea-88c1d97df1e9-kube-api-access-rntrn\") pod \"4931c025-ff04-44cf-80ea-88c1d97df1e9\" (UID: \"4931c025-ff04-44cf-80ea-88c1d97df1e9\") " Oct 30 21:55:31.890745 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:31.890693 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4931c025-ff04-44cf-80ea-88c1d97df1e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4931c025-ff04-44cf-80ea-88c1d97df1e9-maas-api-openshift-ca\") pod \"4931c025-ff04-44cf-80ea-88c1d97df1e9\" (UID: \"4931c025-ff04-44cf-80ea-88c1d97df1e9\") " Oct 30 21:55:31.891495 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:31.891463 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4931c025-ff04-44cf-80ea-88c1d97df1e9-maas-api-openshift-ca" pod "4931c025-ff04-44cf-80ea-88c1d97df1e9" (UID: "4931c025-ff04-44cf-80ea-88c1d97df1e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:31.902076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:31.894028 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4931c025-ff04-44cf-80ea-88c1d97df1e9-maas-api-xks-ca" pod "4931c025-ff04-44cf-80ea-88c1d97df1e9" (UID: "4931c025-ff04-44cf-80ea-88c1d97df1e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:31.902076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:31.894164 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4931c025-ff04-44cf-80ea-88c1d97df1e9-kube-api-access-rntrn" pod "4931c025-ff04-44cf-80ea-88c1d97df1e9" (UID: "4931c025-ff04-44cf-80ea-88c1d97df1e9"). InnerVolumeSpecName "kube-api-access-rntrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:31.992571 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:31.992526 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rntrn\" (UniqueName: \"kubernetes.io/projected/4931c025-ff04-44cf-80ea-88c1d97df1e9-kube-api-access-rntrn\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:31.992571 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:31.992562 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4931c025-ff04-44cf-80ea-88c1d97df1e9-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:31.992571 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:31.992573 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4931c025-ff04-44cf-80ea-88c1d97df1e9-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:32.673822 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:32.673784 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de26f85c-bkrd7" Oct 30 21:55:32.674353 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:32.673826 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de26f85c-bkrd7" event={"ID":"4931c025-ff04-44cf-80ea-88c1d97df1e9","Type":"ContainerDied","Data":"120771f01ba6913f18b98f7194f1c2ddcccf38182d24c7da1a09f3e6d9babd32"} Oct 30 21:55:32.674353 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:32.673935 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="120771f01ba6913f18b98f7194f1c2ddcccf38182d24c7da1a09f3e6d9babd32" Oct 30 21:55:32.864941 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:32.864907 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6" Oct 30 21:55:32.905759 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:32.905723 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c19f7b-d721-4ffa-9b72-a671ea117191-kube-api-access-5kdwx\" (UniqueName: \"kubernetes.io/projected/d0c19f7b-d721-4ffa-9b72-a671ea117191-kube-api-access-5kdwx\") pod \"d0c19f7b-d721-4ffa-9b72-a671ea117191\" (UID: \"d0c19f7b-d721-4ffa-9b72-a671ea117191\") " Oct 30 21:55:32.905933 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:32.905858 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0c19f7b-d721-4ffa-9b72-a671ea117191-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c19f7b-d721-4ffa-9b72-a671ea117191-maas-api-xks-ca\") pod \"d0c19f7b-d721-4ffa-9b72-a671ea117191\" (UID: \"d0c19f7b-d721-4ffa-9b72-a671ea117191\") " Oct 30 21:55:32.905933 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:32.905919 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c19f7b-d721-4ffa-9b72-a671ea117191-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c19f7b-d721-4ffa-9b72-a671ea117191-maas-api-openshift-ca\") pod \"d0c19f7b-d721-4ffa-9b72-a671ea117191\" (UID: \"d0c19f7b-d721-4ffa-9b72-a671ea117191\") " Oct 30 21:55:32.906364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:32.906330 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c19f7b-d721-4ffa-9b72-a671ea117191-maas-api-openshift-ca" pod "d0c19f7b-d721-4ffa-9b72-a671ea117191" (UID: "d0c19f7b-d721-4ffa-9b72-a671ea117191"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:32.906501 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:32.906468 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0c19f7b-d721-4ffa-9b72-a671ea117191-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:32.908005 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:32.907976 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c19f7b-d721-4ffa-9b72-a671ea117191-maas-api-xks-ca" pod "d0c19f7b-d721-4ffa-9b72-a671ea117191" (UID: "d0c19f7b-d721-4ffa-9b72-a671ea117191"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:32.908120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:32.908078 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c19f7b-d721-4ffa-9b72-a671ea117191-kube-api-access-5kdwx" pod "d0c19f7b-d721-4ffa-9b72-a671ea117191" (UID: "d0c19f7b-d721-4ffa-9b72-a671ea117191"). InnerVolumeSpecName "kube-api-access-5kdwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:32.981229 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:32.981144 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9"] Oct 30 21:55:33.008481 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:33.008446 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0c19f7b-d721-4ffa-9b72-a671ea117191-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:33.008481 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:33.008483 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kdwx\" (UniqueName: \"kubernetes.io/projected/d0c19f7b-d721-4ffa-9b72-a671ea117191-kube-api-access-5kdwx\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:33.684858 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:33.684829 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6" Oct 30 21:55:33.685376 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:33.684874 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6" event={"ID":"d0c19f7b-d721-4ffa-9b72-a671ea117191","Type":"ContainerDied","Data":"cb88d4f09400e3b5c5135699644119215e2e916332d81e1510b43a368ca9c71d"} Oct 30 21:55:33.685376 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:33.684936 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb88d4f09400e3b5c5135699644119215e2e916332d81e1510b43a368ca9c71d" Oct 30 21:55:34.160189 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.160053 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t"] Oct 30 21:55:34.160414 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.160293 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" podUID="3d3c5ac1-e264-429e-8ebe-e51f6b5a0038" containerName="maas-api" containerID="cri-o://d3e730c2fab5b035d19fcbfec071bae76ada310c5fb082e7373398b3f8fe6644" gracePeriod=30 Oct 30 21:55:34.525538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.525508 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" Oct 30 21:55:34.630591 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.630555 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-openshift-service-ca\") pod \"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038\" (UID: \"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038\") " Oct 30 21:55:34.630805 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.630645 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-kube-api-access-qhbvp\" (UniqueName: \"kubernetes.io/projected/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-kube-api-access-qhbvp\") pod \"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038\" (UID: \"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038\") " Oct 30 21:55:34.630878 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.630821 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-maas-api-tls\") pod \"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038\" (UID: \"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038\") " Oct 30 21:55:34.630878 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.630846 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-metrics-certs\") pod \"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038\" (UID: \"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038\") " Oct 30 21:55:34.631056 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.631003 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-openshift-service-ca" pod "3d3c5ac1-e264-429e-8ebe-e51f6b5a0038" (UID: "3d3c5ac1-e264-429e-8ebe-e51f6b5a0038"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:34.631190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.631175 4926 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-openshift-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:34.632823 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.632788 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-kube-api-access-qhbvp" pod "3d3c5ac1-e264-429e-8ebe-e51f6b5a0038" (UID: "3d3c5ac1-e264-429e-8ebe-e51f6b5a0038"). InnerVolumeSpecName "kube-api-access-qhbvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:34.632952 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.632892 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-metrics-certs" pod "3d3c5ac1-e264-429e-8ebe-e51f6b5a0038" (UID: "3d3c5ac1-e264-429e-8ebe-e51f6b5a0038"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:34.633046 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.633005 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-maas-api-tls" pod "3d3c5ac1-e264-429e-8ebe-e51f6b5a0038" (UID: "3d3c5ac1-e264-429e-8ebe-e51f6b5a0038"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:34.696022 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.695981 4926 generic.go:356] "Generic (PLEG): container finished" podID="3d3c5ac1-e264-429e-8ebe-e51f6b5a0038" containerID="d3e730c2fab5b035d19fcbfec071bae76ada310c5fb082e7373398b3f8fe6644" exitCode=0 Oct 30 21:55:34.696532 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.696238 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" event={"ID":"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038","Type":"ContainerDied","Data":"d3e730c2fab5b035d19fcbfec071bae76ada310c5fb082e7373398b3f8fe6644"} Oct 30 21:55:34.696532 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.696297 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" event={"ID":"3d3c5ac1-e264-429e-8ebe-e51f6b5a0038","Type":"ContainerDied","Data":"692a4db22d036c00466e9f05207dd1bc8febca5e49fdcbd6f623ed9f69e862be"} Oct 30 21:55:34.696532 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.696330 4926 scope.go:122] "RemoveContainer" containerID="d3e730c2fab5b035d19fcbfec071bae76ada310c5fb082e7373398b3f8fe6644" Oct 30 21:55:34.696532 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.696502 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t" Oct 30 21:55:34.714434 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.714312 4926 scope.go:122] "RemoveContainer" containerID="d3e730c2fab5b035d19fcbfec071bae76ada310c5fb082e7373398b3f8fe6644" Oct 30 21:55:34.715276 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:55:34.715241 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3e730c2fab5b035d19fcbfec071bae76ada310c5fb082e7373398b3f8fe6644\": container with ID starting with d3e730c2fab5b035d19fcbfec071bae76ada310c5fb082e7373398b3f8fe6644 not found: ID does not exist" containerID="d3e730c2fab5b035d19fcbfec071bae76ada310c5fb082e7373398b3f8fe6644" Oct 30 21:55:34.715389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.715286 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3e730c2fab5b035d19fcbfec071bae76ada310c5fb082e7373398b3f8fe6644"} err="failed to get container status \"d3e730c2fab5b035d19fcbfec071bae76ada310c5fb082e7373398b3f8fe6644\": rpc error: code = NotFound desc = could not find container \"d3e730c2fab5b035d19fcbfec071bae76ada310c5fb082e7373398b3f8fe6644\": container with ID starting with d3e730c2fab5b035d19fcbfec071bae76ada310c5fb082e7373398b3f8fe6644 not found: ID does not exist" Oct 30 21:55:34.732648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.732622 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-maas-api-tls\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:34.732648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.732648 4926 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-metrics-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:34.732853 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.732666 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhbvp\" (UniqueName: \"kubernetes.io/projected/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038-kube-api-access-qhbvp\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:34.756538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.756391 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t"] Oct 30 21:55:34.768148 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:34.768108 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e60e16-86d6cc7456-qst5t"] Oct 30 21:55:35.019355 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:35.019269 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d3c5ac1-e264-429e-8ebe-e51f6b5a0038" path="/var/lib/kubelet/pods/3d3c5ac1-e264-429e-8ebe-e51f6b5a0038/volumes" Oct 30 21:55:35.068619 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:35.068586 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v"] Oct 30 21:55:35.075944 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:35.075905 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-e60e16-7hs2v"] Oct 30 21:55:37.016586 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:37.016549 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc62639f-743f-40a1-ae44-d7c132e74fb6" path="/var/lib/kubelet/pods/bc62639f-743f-40a1-ae44-d7c132e74fb6/volumes" Oct 30 21:55:37.559299 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:55:37.558421 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:55:37.981644 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:37.981421 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9" podUID="57902142-df3e-4114-ac6c-7d42a300f2f9" containerName="payload-processing" containerID="cri-o://bd88847d4fd0f826b96b52a1e19ed25743f182c7920f92094816cb9f1eceb002" gracePeriod=25 Oct 30 21:55:39.181761 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.181712 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb"] Oct 30 21:55:39.182538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.182516 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d0c19f7b-d721-4ffa-9b72-a671ea117191" containerName="revoke-keys" Oct 30 21:55:39.182618 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.182539 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c19f7b-d721-4ffa-9b72-a671ea117191" containerName="revoke-keys" Oct 30 21:55:39.182618 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.182565 4926 cpu_manager.go:414] "containerMap: removing container" podUID="4931c025-ff04-44cf-80ea-88c1d97df1e9" containerName="revoke-keys" Oct 30 21:55:39.182618 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.182576 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4931c025-ff04-44cf-80ea-88c1d97df1e9" containerName="revoke-keys" Oct 30 21:55:39.182618 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.182596 4926 cpu_manager.go:414] "containerMap: removing container" podUID="3d3c5ac1-e264-429e-8ebe-e51f6b5a0038" containerName="maas-api" Oct 30 21:55:39.182618 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.182604 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3c5ac1-e264-429e-8ebe-e51f6b5a0038" containerName="maas-api" Oct 30 21:55:39.182888 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.182773 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c19f7b-d721-4ffa-9b72-a671ea117191" containerName="revoke-keys" Oct 30 21:55:39.182888 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.182786 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="4931c025-ff04-44cf-80ea-88c1d97df1e9" containerName="revoke-keys" Oct 30 21:55:39.182888 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.182798 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3c5ac1-e264-429e-8ebe-e51f6b5a0038" containerName="maas-api" Oct 30 21:55:39.188613 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.188587 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb" Oct 30 21:55:39.190366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.190332 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9" podUID="57902142-df3e-4114-ac6c-7d42a300f2f9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:55:39.206923 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.206892 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb"] Oct 30 21:55:39.208830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.208801 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:55:39.209070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.209013 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:55:39.261118 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.261083 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:55:39.261299 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.261278 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:55:39.303994 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.303953 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pczpc\" (UniqueName: \"kubernetes.io/projected/69f0ac4f-81c8-4648-9b24-2d15b8720d41-kube-api-access-pczpc\") pod \"maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb\" (UID: \"69f0ac4f-81c8-4648-9b24-2d15b8720d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb" Oct 30 21:55:39.304222 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.304018 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f0ac4f-81c8-4648-9b24-2d15b8720d41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb\" (UID: \"69f0ac4f-81c8-4648-9b24-2d15b8720d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb" Oct 30 21:55:39.304222 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.304167 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f0ac4f-81c8-4648-9b24-2d15b8720d41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb\" (UID: \"69f0ac4f-81c8-4648-9b24-2d15b8720d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb" Oct 30 21:55:39.405781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.405740 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pczpc\" (UniqueName: \"kubernetes.io/projected/69f0ac4f-81c8-4648-9b24-2d15b8720d41-kube-api-access-pczpc\") pod \"maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb\" (UID: \"69f0ac4f-81c8-4648-9b24-2d15b8720d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb" Oct 30 21:55:39.406018 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.405796 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f0ac4f-81c8-4648-9b24-2d15b8720d41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb\" (UID: \"69f0ac4f-81c8-4648-9b24-2d15b8720d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb" Oct 30 21:55:39.406018 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.405830 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f0ac4f-81c8-4648-9b24-2d15b8720d41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb\" (UID: \"69f0ac4f-81c8-4648-9b24-2d15b8720d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb" Oct 30 21:55:39.406553 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.406527 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f0ac4f-81c8-4648-9b24-2d15b8720d41-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb\" (UID: \"69f0ac4f-81c8-4648-9b24-2d15b8720d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb" Oct 30 21:55:39.408391 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.408366 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f0ac4f-81c8-4648-9b24-2d15b8720d41-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb\" (UID: \"69f0ac4f-81c8-4648-9b24-2d15b8720d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb" Oct 30 21:55:39.427270 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.427239 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pczpc\" (UniqueName: \"kubernetes.io/projected/69f0ac4f-81c8-4648-9b24-2d15b8720d41-kube-api-access-pczpc\") pod \"maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb\" (UID: \"69f0ac4f-81c8-4648-9b24-2d15b8720d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb" Oct 30 21:55:39.508818 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.508726 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb" Oct 30 21:55:39.812787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:39.812755 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb"] Oct 30 21:55:39.814562 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:55:39.814528 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f0ac4f_81c8_4648_9b24_2d15b8720d41.slice/crio-a12f97566a1cf786f466266f0e14f6dfc17865eda9b3fc68ebfdd1bd6415b7a5 WatchSource:0}: Error finding container a12f97566a1cf786f466266f0e14f6dfc17865eda9b3fc68ebfdd1bd6415b7a5: Status 404 returned error can't find the container with id a12f97566a1cf786f466266f0e14f6dfc17865eda9b3fc68ebfdd1bd6415b7a5 Oct 30 21:55:40.803119 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:40.803018 4926 generic.go:356] "Generic (PLEG): container finished" podID="69f0ac4f-81c8-4648-9b24-2d15b8720d41" containerID="605d123d66bf4a7b7cbd11d77ffc26e8096b26a6112361b80bb4b79d1728acbf" exitCode=0 Oct 30 21:55:40.803502 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:40.803146 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb" event={"ID":"69f0ac4f-81c8-4648-9b24-2d15b8720d41","Type":"ContainerDied","Data":"605d123d66bf4a7b7cbd11d77ffc26e8096b26a6112361b80bb4b79d1728acbf"} Oct 30 21:55:40.803502 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:40.803183 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb" event={"ID":"69f0ac4f-81c8-4648-9b24-2d15b8720d41","Type":"ContainerStarted","Data":"a12f97566a1cf786f466266f0e14f6dfc17865eda9b3fc68ebfdd1bd6415b7a5"} Oct 30 21:55:42.000841 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:42.000807 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb" Oct 30 21:55:42.141014 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:42.140928 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69f0ac4f-81c8-4648-9b24-2d15b8720d41-kube-api-access-pczpc\" (UniqueName: \"kubernetes.io/projected/69f0ac4f-81c8-4648-9b24-2d15b8720d41-kube-api-access-pczpc\") pod \"69f0ac4f-81c8-4648-9b24-2d15b8720d41\" (UID: \"69f0ac4f-81c8-4648-9b24-2d15b8720d41\") " Oct 30 21:55:42.141201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:42.141085 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69f0ac4f-81c8-4648-9b24-2d15b8720d41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f0ac4f-81c8-4648-9b24-2d15b8720d41-maas-api-openshift-ca\") pod \"69f0ac4f-81c8-4648-9b24-2d15b8720d41\" (UID: \"69f0ac4f-81c8-4648-9b24-2d15b8720d41\") " Oct 30 21:55:42.141201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:42.141182 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69f0ac4f-81c8-4648-9b24-2d15b8720d41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f0ac4f-81c8-4648-9b24-2d15b8720d41-maas-api-xks-ca\") pod \"69f0ac4f-81c8-4648-9b24-2d15b8720d41\" (UID: \"69f0ac4f-81c8-4648-9b24-2d15b8720d41\") " Oct 30 21:55:42.141943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:42.141784 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f0ac4f-81c8-4648-9b24-2d15b8720d41-maas-api-openshift-ca" pod "69f0ac4f-81c8-4648-9b24-2d15b8720d41" (UID: "69f0ac4f-81c8-4648-9b24-2d15b8720d41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:42.143866 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:42.142632 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69f0ac4f-81c8-4648-9b24-2d15b8720d41-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:42.145318 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:42.144524 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f0ac4f-81c8-4648-9b24-2d15b8720d41-kube-api-access-pczpc" pod "69f0ac4f-81c8-4648-9b24-2d15b8720d41" (UID: "69f0ac4f-81c8-4648-9b24-2d15b8720d41"). InnerVolumeSpecName "kube-api-access-pczpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:42.153928 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:42.153882 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69f0ac4f-81c8-4648-9b24-2d15b8720d41-maas-api-xks-ca" pod "69f0ac4f-81c8-4648-9b24-2d15b8720d41" (UID: "69f0ac4f-81c8-4648-9b24-2d15b8720d41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:42.243699 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:42.243659 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pczpc\" (UniqueName: \"kubernetes.io/projected/69f0ac4f-81c8-4648-9b24-2d15b8720d41-kube-api-access-pczpc\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:42.243699 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:42.243694 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69f0ac4f-81c8-4648-9b24-2d15b8720d41-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:42.825701 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:42.824840 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb" Oct 30 21:55:42.825701 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:42.824872 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb" event={"ID":"69f0ac4f-81c8-4648-9b24-2d15b8720d41","Type":"ContainerDied","Data":"a12f97566a1cf786f466266f0e14f6dfc17865eda9b3fc68ebfdd1bd6415b7a5"} Oct 30 21:55:42.825701 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:42.825028 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a12f97566a1cf786f466266f0e14f6dfc17865eda9b3fc68ebfdd1bd6415b7a5" Oct 30 21:55:42.877758 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:42.877724 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56"] Oct 30 21:55:42.877986 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:42.877946 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" podUID="9311b3e6-ceb2-49dd-ac53-dca966a540ce" containerName="istio-proxy" containerID="cri-o://d050528c0af19898255956875d89a95870917075d4b657bffa3f09c6479a0c17" gracePeriod=30 Oct 30 21:55:43.711104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:43.711063 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px"] Oct 30 21:55:43.711876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:43.711852 4926 cpu_manager.go:414] "containerMap: removing container" podUID="69f0ac4f-81c8-4648-9b24-2d15b8720d41" containerName="revoke-keys" Oct 30 21:55:43.711940 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:43.711877 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69f0ac4f-81c8-4648-9b24-2d15b8720d41" containerName="revoke-keys" Oct 30 21:55:43.712006 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:43.711996 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="69f0ac4f-81c8-4648-9b24-2d15b8720d41" containerName="revoke-keys" Oct 30 21:55:43.717975 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:43.717955 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px" Oct 30 21:55:43.741298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:43.741267 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px"] Oct 30 21:55:43.747783 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:43.747756 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:55:43.747956 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:43.747781 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:55:43.769877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:43.769837 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsqj6\" (UniqueName: \"kubernetes.io/projected/ab95c2d5-2782-4264-85bc-61e70bdc7a00-kube-api-access-vsqj6\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px\" (UID: \"ab95c2d5-2782-4264-85bc-61e70bdc7a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px" Oct 30 21:55:43.774021 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:43.772379 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab95c2d5-2782-4264-85bc-61e70bdc7a00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px\" (UID: \"ab95c2d5-2782-4264-85bc-61e70bdc7a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px" Oct 30 21:55:43.774021 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:43.772564 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab95c2d5-2782-4264-85bc-61e70bdc7a00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px\" (UID: \"ab95c2d5-2782-4264-85bc-61e70bdc7a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px" Oct 30 21:55:43.780208 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:43.780169 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:55:43.780386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:43.780327 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:55:43.873700 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:43.873658 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsqj6\" (UniqueName: \"kubernetes.io/projected/ab95c2d5-2782-4264-85bc-61e70bdc7a00-kube-api-access-vsqj6\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px\" (UID: \"ab95c2d5-2782-4264-85bc-61e70bdc7a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px" Oct 30 21:55:43.873931 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:43.873717 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab95c2d5-2782-4264-85bc-61e70bdc7a00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px\" (UID: \"ab95c2d5-2782-4264-85bc-61e70bdc7a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px" Oct 30 21:55:43.873931 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:43.873764 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab95c2d5-2782-4264-85bc-61e70bdc7a00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px\" (UID: \"ab95c2d5-2782-4264-85bc-61e70bdc7a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px" Oct 30 21:55:43.874440 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:43.874412 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab95c2d5-2782-4264-85bc-61e70bdc7a00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px\" (UID: \"ab95c2d5-2782-4264-85bc-61e70bdc7a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px" Oct 30 21:55:43.876182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:43.876158 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab95c2d5-2782-4264-85bc-61e70bdc7a00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px\" (UID: \"ab95c2d5-2782-4264-85bc-61e70bdc7a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px" Oct 30 21:55:43.894975 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:43.894947 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsqj6\" (UniqueName: \"kubernetes.io/projected/ab95c2d5-2782-4264-85bc-61e70bdc7a00-kube-api-access-vsqj6\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px\" (UID: \"ab95c2d5-2782-4264-85bc-61e70bdc7a00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px" Oct 30 21:55:44.033331 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:44.033256 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px" Oct 30 21:55:44.348294 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:44.348258 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px"] Oct 30 21:55:44.348573 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:55:44.348542 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab95c2d5_2782_4264_85bc_61e70bdc7a00.slice/crio-10b24f8c3819c6fad1c981ae84c0413e63488abf1c22b0189d6988e816eec893 WatchSource:0}: Error finding container 10b24f8c3819c6fad1c981ae84c0413e63488abf1c22b0189d6988e816eec893: Status 404 returned error can't find the container with id 10b24f8c3819c6fad1c981ae84c0413e63488abf1c22b0189d6988e816eec893 Oct 30 21:55:44.842412 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:44.842366 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px" event={"ID":"ab95c2d5-2782-4264-85bc-61e70bdc7a00","Type":"ContainerStarted","Data":"1f936626315eabd5982f8dc3aed39bae42c2531b86e7054f63bb5e8636549625"} Oct 30 21:55:44.842889 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:44.842426 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px" event={"ID":"ab95c2d5-2782-4264-85bc-61e70bdc7a00","Type":"ContainerStarted","Data":"10b24f8c3819c6fad1c981ae84c0413e63488abf1c22b0189d6988e816eec893"} Oct 30 21:55:44.890198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:44.890136 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px" podStartSLOduration=1.8901172050000001 podStartE2EDuration="1.890117205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:44.88606279 +0000 UTC m=+3332.007136536" watchObservedRunningTime="2026-10-30 21:55:44.890117205 +0000 UTC m=+3332.011190954" Oct 30 21:55:45.853525 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:45.853482 4926 generic.go:356] "Generic (PLEG): container finished" podID="ab95c2d5-2782-4264-85bc-61e70bdc7a00" containerID="1f936626315eabd5982f8dc3aed39bae42c2531b86e7054f63bb5e8636549625" exitCode=0 Oct 30 21:55:45.853983 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:45.853572 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px" event={"ID":"ab95c2d5-2782-4264-85bc-61e70bdc7a00","Type":"ContainerDied","Data":"1f936626315eabd5982f8dc3aed39bae42c2531b86e7054f63bb5e8636549625"} Oct 30 21:55:47.046947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:47.046919 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px" Oct 30 21:55:47.113570 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:47.113482 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab95c2d5-2782-4264-85bc-61e70bdc7a00-kube-api-access-vsqj6\" (UniqueName: \"kubernetes.io/projected/ab95c2d5-2782-4264-85bc-61e70bdc7a00-kube-api-access-vsqj6\") pod \"ab95c2d5-2782-4264-85bc-61e70bdc7a00\" (UID: \"ab95c2d5-2782-4264-85bc-61e70bdc7a00\") " Oct 30 21:55:47.113759 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:47.113634 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab95c2d5-2782-4264-85bc-61e70bdc7a00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab95c2d5-2782-4264-85bc-61e70bdc7a00-maas-api-xks-ca\") pod \"ab95c2d5-2782-4264-85bc-61e70bdc7a00\" (UID: \"ab95c2d5-2782-4264-85bc-61e70bdc7a00\") " Oct 30 21:55:47.113909 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:47.113862 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab95c2d5-2782-4264-85bc-61e70bdc7a00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab95c2d5-2782-4264-85bc-61e70bdc7a00-maas-api-openshift-ca\") pod \"ab95c2d5-2782-4264-85bc-61e70bdc7a00\" (UID: \"ab95c2d5-2782-4264-85bc-61e70bdc7a00\") " Oct 30 21:55:47.114686 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:47.114415 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab95c2d5-2782-4264-85bc-61e70bdc7a00-maas-api-openshift-ca" pod "ab95c2d5-2782-4264-85bc-61e70bdc7a00" (UID: "ab95c2d5-2782-4264-85bc-61e70bdc7a00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:47.116204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:47.116177 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab95c2d5-2782-4264-85bc-61e70bdc7a00-kube-api-access-vsqj6" pod "ab95c2d5-2782-4264-85bc-61e70bdc7a00" (UID: "ab95c2d5-2782-4264-85bc-61e70bdc7a00"). InnerVolumeSpecName "kube-api-access-vsqj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:47.116717 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:47.116675 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab95c2d5-2782-4264-85bc-61e70bdc7a00-maas-api-xks-ca" pod "ab95c2d5-2782-4264-85bc-61e70bdc7a00" (UID: "ab95c2d5-2782-4264-85bc-61e70bdc7a00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:55:47.215337 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:47.215299 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab95c2d5-2782-4264-85bc-61e70bdc7a00-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:47.215337 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:47.215330 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab95c2d5-2782-4264-85bc-61e70bdc7a00-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:47.215337 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:47.215344 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsqj6\" (UniqueName: \"kubernetes.io/projected/ab95c2d5-2782-4264-85bc-61e70bdc7a00-kube-api-access-vsqj6\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:47.937692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:47.935628 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px" event={"ID":"ab95c2d5-2782-4264-85bc-61e70bdc7a00","Type":"ContainerDied","Data":"10b24f8c3819c6fad1c981ae84c0413e63488abf1c22b0189d6988e816eec893"} Oct 30 21:55:47.937692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:47.935697 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10b24f8c3819c6fad1c981ae84c0413e63488abf1c22b0189d6988e816eec893" Oct 30 21:55:47.937692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:47.935775 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px" Oct 30 21:55:48.161306 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.161241 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:55:48.234462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.234422 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istiod-ca-cert\") pod \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " Oct 30 21:55:48.234672 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.234559 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-token\" (UniqueName: \"kubernetes.io/projected/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-token\") pod \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " Oct 30 21:55:48.234672 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.234640 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-credential-socket\") pod \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " Oct 30 21:55:48.234803 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.234678 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-workload-certs\") pod \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " Oct 30 21:55:48.234803 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.234724 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-workload-socket\") pod \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " Oct 30 21:55:48.234803 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.234764 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-podinfo\") pod \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " Oct 30 21:55:48.234976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.234806 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-data\") pod \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " Oct 30 21:55:48.234976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.234865 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istiod-ca-cert" pod "9311b3e6-ceb2-49dd-ac53-dca966a540ce" (UID: "9311b3e6-ceb2-49dd-ac53-dca966a540ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:48.234976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.234874 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-envoy\") pod \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " Oct 30 21:55:48.234976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.234947 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9311b3e6-ceb2-49dd-ac53-dca966a540ce-kube-api-access-wfw4x\" (UniqueName: \"kubernetes.io/projected/9311b3e6-ceb2-49dd-ac53-dca966a540ce-kube-api-access-wfw4x\") pod \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\" (UID: \"9311b3e6-ceb2-49dd-ac53-dca966a540ce\") " Oct 30 21:55:48.235880 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.235439 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-data" pod "9311b3e6-ceb2-49dd-ac53-dca966a540ce" (UID: "9311b3e6-ceb2-49dd-ac53-dca966a540ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:48.235880 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.235599 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-credential-socket" pod "9311b3e6-ceb2-49dd-ac53-dca966a540ce" (UID: "9311b3e6-ceb2-49dd-ac53-dca966a540ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:48.235880 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.235667 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-workload-socket" pod "9311b3e6-ceb2-49dd-ac53-dca966a540ce" (UID: "9311b3e6-ceb2-49dd-ac53-dca966a540ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:48.235880 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.235761 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:48.235880 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.235781 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:48.235880 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.235794 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:48.235880 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.235850 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-workload-certs" pod "9311b3e6-ceb2-49dd-ac53-dca966a540ce" (UID: "9311b3e6-ceb2-49dd-ac53-dca966a540ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:48.242070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.237541 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-envoy" pod "9311b3e6-ceb2-49dd-ac53-dca966a540ce" (UID: "9311b3e6-ceb2-49dd-ac53-dca966a540ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:55:48.242070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.237823 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-token" pod "9311b3e6-ceb2-49dd-ac53-dca966a540ce" (UID: "9311b3e6-ceb2-49dd-ac53-dca966a540ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:48.242070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.237925 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9311b3e6-ceb2-49dd-ac53-dca966a540ce-kube-api-access-wfw4x" pod "9311b3e6-ceb2-49dd-ac53-dca966a540ce" (UID: "9311b3e6-ceb2-49dd-ac53-dca966a540ce"). InnerVolumeSpecName "kube-api-access-wfw4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:48.242070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.237947 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-podinfo" pod "9311b3e6-ceb2-49dd-ac53-dca966a540ce" (UID: "9311b3e6-ceb2-49dd-ac53-dca966a540ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:55:48.337107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.337064 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:48.337107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.337105 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfw4x\" (UniqueName: \"kubernetes.io/projected/9311b3e6-ceb2-49dd-ac53-dca966a540ce-kube-api-access-wfw4x\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:48.337396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.337120 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:48.337396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.337133 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:48.337396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.337144 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9311b3e6-ceb2-49dd-ac53-dca966a540ce-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:48.337396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.337157 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9311b3e6-ceb2-49dd-ac53-dca966a540ce-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:48.339556 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.339516 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s"] Oct 30 21:55:48.343487 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.343455 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9cd83f-b6q4s"] Oct 30 21:55:48.946169 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.946131 4926 generic.go:356] "Generic (PLEG): container finished" podID="9311b3e6-ceb2-49dd-ac53-dca966a540ce" containerID="d050528c0af19898255956875d89a95870917075d4b657bffa3f09c6479a0c17" exitCode=0 Oct 30 21:55:48.946353 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.946177 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" Oct 30 21:55:48.946353 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.946209 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" event={"ID":"9311b3e6-ceb2-49dd-ac53-dca966a540ce","Type":"ContainerDied","Data":"d050528c0af19898255956875d89a95870917075d4b657bffa3f09c6479a0c17"} Oct 30 21:55:48.946353 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.946269 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56" event={"ID":"9311b3e6-ceb2-49dd-ac53-dca966a540ce","Type":"ContainerDied","Data":"f169d6b40b7a11131aced943c57cc06e5a02e472359954e96fb71c4710239570"} Oct 30 21:55:48.946353 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.946292 4926 scope.go:122] "RemoveContainer" containerID="d050528c0af19898255956875d89a95870917075d4b657bffa3f09c6479a0c17" Oct 30 21:55:48.959378 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.959361 4926 scope.go:122] "RemoveContainer" containerID="806e121a53bd76a9ee9392323b72eb5d0a3cf75d7ff232969ce4085224a4ba55" Oct 30 21:55:48.970103 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.970079 4926 scope.go:122] "RemoveContainer" containerID="d050528c0af19898255956875d89a95870917075d4b657bffa3f09c6479a0c17" Oct 30 21:55:48.970389 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:55:48.970372 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d050528c0af19898255956875d89a95870917075d4b657bffa3f09c6479a0c17\": container with ID starting with d050528c0af19898255956875d89a95870917075d4b657bffa3f09c6479a0c17 not found: ID does not exist" containerID="d050528c0af19898255956875d89a95870917075d4b657bffa3f09c6479a0c17" Oct 30 21:55:48.970436 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.970398 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d050528c0af19898255956875d89a95870917075d4b657bffa3f09c6479a0c17"} err="failed to get container status \"d050528c0af19898255956875d89a95870917075d4b657bffa3f09c6479a0c17\": rpc error: code = NotFound desc = could not find container \"d050528c0af19898255956875d89a95870917075d4b657bffa3f09c6479a0c17\": container with ID starting with d050528c0af19898255956875d89a95870917075d4b657bffa3f09c6479a0c17 not found: ID does not exist" Oct 30 21:55:48.970436 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.970418 4926 scope.go:122] "RemoveContainer" containerID="806e121a53bd76a9ee9392323b72eb5d0a3cf75d7ff232969ce4085224a4ba55" Oct 30 21:55:48.970660 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:55:48.970643 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"806e121a53bd76a9ee9392323b72eb5d0a3cf75d7ff232969ce4085224a4ba55\": container with ID starting with 806e121a53bd76a9ee9392323b72eb5d0a3cf75d7ff232969ce4085224a4ba55 not found: ID does not exist" containerID="806e121a53bd76a9ee9392323b72eb5d0a3cf75d7ff232969ce4085224a4ba55" Oct 30 21:55:48.970716 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:48.970666 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"806e121a53bd76a9ee9392323b72eb5d0a3cf75d7ff232969ce4085224a4ba55"} err="failed to get container status \"806e121a53bd76a9ee9392323b72eb5d0a3cf75d7ff232969ce4085224a4ba55\": rpc error: code = NotFound desc = could not find container \"806e121a53bd76a9ee9392323b72eb5d0a3cf75d7ff232969ce4085224a4ba55\": container with ID starting with 806e121a53bd76a9ee9392323b72eb5d0a3cf75d7ff232969ce4085224a4ba55 not found: ID does not exist" Oct 30 21:55:49.007523 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:49.007490 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56"] Oct 30 21:55:49.016500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:49.016465 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af1cde02-c2dd-4dfb-8596-ba673bfcb5ec" path="/var/lib/kubelet/pods/af1cde02-c2dd-4dfb-8596-ba673bfcb5ec/volumes" Oct 30 21:55:49.016976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:49.016957 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-e60e16-openshift-default-765bfc6d7f-slk56"] Oct 30 21:55:49.190976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:49.190938 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9" podUID="57902142-df3e-4114-ac6c-7d42a300f2f9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:55:51.018190 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:51.018154 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9311b3e6-ceb2-49dd-ac53-dca966a540ce" path="/var/lib/kubelet/pods/9311b3e6-ceb2-49dd-ac53-dca966a540ce/volumes" Oct 30 21:55:53.879732 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:53.879695 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j"] Oct 30 21:55:53.889943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:53.889895 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-889195-gvn8j"] Oct 30 21:55:55.015341 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:55.015296 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88d0b36e-2321-4b48-a80b-9428ef221151" path="/var/lib/kubelet/pods/88d0b36e-2321-4b48-a80b-9428ef221151/volumes" Oct 30 21:55:56.394789 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:56.394742 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5"] Oct 30 21:55:56.395752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:56.395714 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" podUID="44fcef78-1e16-4d54-9306-52f006c0bb9f" containerName="istio-proxy" containerID="cri-o://0e29aabd94a498c95d50dd7e7fc44bb7fa88cf6d648357afaf19538380a24582" gracePeriod=30 Oct 30 21:55:57.220341 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.220297 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng"] Oct 30 21:55:57.220968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.220945 4926 cpu_manager.go:414] "containerMap: removing container" podUID="ab95c2d5-2782-4264-85bc-61e70bdc7a00" containerName="revoke-keys" Oct 30 21:55:57.220968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.220966 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab95c2d5-2782-4264-85bc-61e70bdc7a00" containerName="revoke-keys" Oct 30 21:55:57.221176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.220977 4926 cpu_manager.go:414] "containerMap: removing container" podUID="9311b3e6-ceb2-49dd-ac53-dca966a540ce" containerName="istio-proxy" Oct 30 21:55:57.221176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.220982 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9311b3e6-ceb2-49dd-ac53-dca966a540ce" containerName="istio-proxy" Oct 30 21:55:57.221176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.220990 4926 cpu_manager.go:414] "containerMap: removing container" podUID="9311b3e6-ceb2-49dd-ac53-dca966a540ce" containerName="istio-proxy" Oct 30 21:55:57.221176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.220996 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9311b3e6-ceb2-49dd-ac53-dca966a540ce" containerName="istio-proxy" Oct 30 21:55:57.221176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.221140 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab95c2d5-2782-4264-85bc-61e70bdc7a00" containerName="revoke-keys" Oct 30 21:55:57.221176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.221157 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="9311b3e6-ceb2-49dd-ac53-dca966a540ce" containerName="istio-proxy" Oct 30 21:55:57.221176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.221169 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="9311b3e6-ceb2-49dd-ac53-dca966a540ce" containerName="istio-proxy" Oct 30 21:55:57.226852 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.226827 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng" Oct 30 21:55:57.257693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.257603 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:55:57.257693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.257617 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:55:57.257693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.257617 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:55:57.257693 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.257666 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:55:57.262762 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.262738 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng"] Oct 30 21:55:57.355818 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.355782 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/170ffbae-db1d-4d75-bc42-6548ea5c74bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng\" (UID: \"170ffbae-db1d-4d75-bc42-6548ea5c74bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng" Oct 30 21:55:57.356013 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.355983 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/170ffbae-db1d-4d75-bc42-6548ea5c74bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng\" (UID: \"170ffbae-db1d-4d75-bc42-6548ea5c74bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng" Oct 30 21:55:57.356107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.356077 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkgzv\" (UniqueName: \"kubernetes.io/projected/170ffbae-db1d-4d75-bc42-6548ea5c74bc-kube-api-access-zkgzv\") pod \"maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng\" (UID: \"170ffbae-db1d-4d75-bc42-6548ea5c74bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng" Oct 30 21:55:57.457515 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.457482 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/170ffbae-db1d-4d75-bc42-6548ea5c74bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng\" (UID: \"170ffbae-db1d-4d75-bc42-6548ea5c74bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng" Oct 30 21:55:57.458008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.457532 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkgzv\" (UniqueName: \"kubernetes.io/projected/170ffbae-db1d-4d75-bc42-6548ea5c74bc-kube-api-access-zkgzv\") pod \"maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng\" (UID: \"170ffbae-db1d-4d75-bc42-6548ea5c74bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng" Oct 30 21:55:57.458008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.457602 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/170ffbae-db1d-4d75-bc42-6548ea5c74bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng\" (UID: \"170ffbae-db1d-4d75-bc42-6548ea5c74bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng" Oct 30 21:55:57.458352 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.458329 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/170ffbae-db1d-4d75-bc42-6548ea5c74bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng\" (UID: \"170ffbae-db1d-4d75-bc42-6548ea5c74bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng" Oct 30 21:55:57.460179 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.460158 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/170ffbae-db1d-4d75-bc42-6548ea5c74bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng\" (UID: \"170ffbae-db1d-4d75-bc42-6548ea5c74bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng" Oct 30 21:55:57.481974 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.481902 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkgzv\" (UniqueName: \"kubernetes.io/projected/170ffbae-db1d-4d75-bc42-6548ea5c74bc-kube-api-access-zkgzv\") pod \"maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng\" (UID: \"170ffbae-db1d-4d75-bc42-6548ea5c74bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng" Oct 30 21:55:57.540314 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.540279 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng" Oct 30 21:55:57.861139 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:57.861016 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng"] Oct 30 21:55:57.864628 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:55:57.864586 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod170ffbae_db1d_4d75_bc42_6548ea5c74bc.slice/crio-a3225f836892b85b640825fe9221577a891afaa19bf717ced981c96739ae9b6a WatchSource:0}: Error finding container a3225f836892b85b640825fe9221577a891afaa19bf717ced981c96739ae9b6a: Status 404 returned error can't find the container with id a3225f836892b85b640825fe9221577a891afaa19bf717ced981c96739ae9b6a Oct 30 21:55:58.034215 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:58.034174 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng" event={"ID":"170ffbae-db1d-4d75-bc42-6548ea5c74bc","Type":"ContainerStarted","Data":"a3225f836892b85b640825fe9221577a891afaa19bf717ced981c96739ae9b6a"} Oct 30 21:55:58.307404 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:58.307375 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9" Oct 30 21:55:58.373714 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:58.373686 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57902142-df3e-4114-ac6c-7d42a300f2f9-kube-api-access-s4vjq\" (UniqueName: \"kubernetes.io/projected/57902142-df3e-4114-ac6c-7d42a300f2f9-kube-api-access-s4vjq\") pod \"57902142-df3e-4114-ac6c-7d42a300f2f9\" (UID: \"57902142-df3e-4114-ac6c-7d42a300f2f9\") " Oct 30 21:55:58.374108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:58.374050 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57902142-df3e-4114-ac6c-7d42a300f2f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57902142-df3e-4114-ac6c-7d42a300f2f9-plugins-config-volume\") pod \"57902142-df3e-4114-ac6c-7d42a300f2f9\" (UID: \"57902142-df3e-4114-ac6c-7d42a300f2f9\") " Oct 30 21:55:58.375131 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:58.375103 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57902142-df3e-4114-ac6c-7d42a300f2f9-plugins-config-volume" pod "57902142-df3e-4114-ac6c-7d42a300f2f9" (UID: "57902142-df3e-4114-ac6c-7d42a300f2f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:55:58.382088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:58.377580 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57902142-df3e-4114-ac6c-7d42a300f2f9-kube-api-access-s4vjq" pod "57902142-df3e-4114-ac6c-7d42a300f2f9" (UID: "57902142-df3e-4114-ac6c-7d42a300f2f9"). InnerVolumeSpecName "kube-api-access-s4vjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:55:58.476002 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:58.475964 4926 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/57902142-df3e-4114-ac6c-7d42a300f2f9-plugins-config-volume\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:58.476449 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:58.476004 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4vjq\" (UniqueName: \"kubernetes.io/projected/57902142-df3e-4114-ac6c-7d42a300f2f9-kube-api-access-s4vjq\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:55:58.497277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:58.497239 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk"] Oct 30 21:55:58.503109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:58.503072 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-79477e-scnvk"] Oct 30 21:55:59.017369 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:59.017338 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c344af02-b407-421d-aea3-6f3789ee9017" path="/var/lib/kubelet/pods/c344af02-b407-421d-aea3-6f3789ee9017/volumes" Oct 30 21:55:59.044635 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:59.044600 4926 generic.go:356] "Generic (PLEG): container finished" podID="170ffbae-db1d-4d75-bc42-6548ea5c74bc" containerID="27c51ee2fed58c25039984d2ee7d66316d6886a721eb705c377f6f0987243fed" exitCode=0 Oct 30 21:55:59.044830 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:59.044687 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng" event={"ID":"170ffbae-db1d-4d75-bc42-6548ea5c74bc","Type":"ContainerDied","Data":"27c51ee2fed58c25039984d2ee7d66316d6886a721eb705c377f6f0987243fed"} Oct 30 21:55:59.046371 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:59.046347 4926 generic.go:356] "Generic (PLEG): container finished" podID="57902142-df3e-4114-ac6c-7d42a300f2f9" containerID="bd88847d4fd0f826b96b52a1e19ed25743f182c7920f92094816cb9f1eceb002" exitCode=0 Oct 30 21:55:59.046483 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:59.046391 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9" Oct 30 21:55:59.046483 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:59.046427 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9" event={"ID":"57902142-df3e-4114-ac6c-7d42a300f2f9","Type":"ContainerDied","Data":"bd88847d4fd0f826b96b52a1e19ed25743f182c7920f92094816cb9f1eceb002"} Oct 30 21:55:59.046483 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:59.046473 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9" event={"ID":"57902142-df3e-4114-ac6c-7d42a300f2f9","Type":"ContainerDied","Data":"80284b8323e2ebba43f4f5115ae76f6de225502e541039dc3948f17ab3c02061"} Oct 30 21:55:59.046599 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:59.046499 4926 scope.go:122] "RemoveContainer" containerID="bd88847d4fd0f826b96b52a1e19ed25743f182c7920f92094816cb9f1eceb002" Oct 30 21:55:59.064251 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:59.064231 4926 scope.go:122] "RemoveContainer" containerID="bd88847d4fd0f826b96b52a1e19ed25743f182c7920f92094816cb9f1eceb002" Oct 30 21:55:59.064556 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:55:59.064530 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd88847d4fd0f826b96b52a1e19ed25743f182c7920f92094816cb9f1eceb002\": container with ID starting with bd88847d4fd0f826b96b52a1e19ed25743f182c7920f92094816cb9f1eceb002 not found: ID does not exist" containerID="bd88847d4fd0f826b96b52a1e19ed25743f182c7920f92094816cb9f1eceb002" Oct 30 21:55:59.064644 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:59.064564 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd88847d4fd0f826b96b52a1e19ed25743f182c7920f92094816cb9f1eceb002"} err="failed to get container status \"bd88847d4fd0f826b96b52a1e19ed25743f182c7920f92094816cb9f1eceb002\": rpc error: code = NotFound desc = could not find container \"bd88847d4fd0f826b96b52a1e19ed25743f182c7920f92094816cb9f1eceb002\": container with ID starting with bd88847d4fd0f826b96b52a1e19ed25743f182c7920f92094816cb9f1eceb002 not found: ID does not exist" Oct 30 21:55:59.137836 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:59.137794 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9"] Oct 30 21:55:59.141289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:55:59.141261 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-889195-66b9c594bd-tpws9"] Oct 30 21:56:00.239959 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:00.239928 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng" Oct 30 21:56:00.297346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:00.297306 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/170ffbae-db1d-4d75-bc42-6548ea5c74bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/170ffbae-db1d-4d75-bc42-6548ea5c74bc-maas-api-xks-ca\") pod \"170ffbae-db1d-4d75-bc42-6548ea5c74bc\" (UID: \"170ffbae-db1d-4d75-bc42-6548ea5c74bc\") " Oct 30 21:56:00.297558 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:00.297459 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/170ffbae-db1d-4d75-bc42-6548ea5c74bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/170ffbae-db1d-4d75-bc42-6548ea5c74bc-maas-api-openshift-ca\") pod \"170ffbae-db1d-4d75-bc42-6548ea5c74bc\" (UID: \"170ffbae-db1d-4d75-bc42-6548ea5c74bc\") " Oct 30 21:56:00.297630 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:00.297596 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/170ffbae-db1d-4d75-bc42-6548ea5c74bc-kube-api-access-zkgzv\" (UniqueName: \"kubernetes.io/projected/170ffbae-db1d-4d75-bc42-6548ea5c74bc-kube-api-access-zkgzv\") pod \"170ffbae-db1d-4d75-bc42-6548ea5c74bc\" (UID: \"170ffbae-db1d-4d75-bc42-6548ea5c74bc\") " Oct 30 21:56:00.298135 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:00.298103 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/170ffbae-db1d-4d75-bc42-6548ea5c74bc-maas-api-openshift-ca" pod "170ffbae-db1d-4d75-bc42-6548ea5c74bc" (UID: "170ffbae-db1d-4d75-bc42-6548ea5c74bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:00.300975 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:00.300947 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/170ffbae-db1d-4d75-bc42-6548ea5c74bc-kube-api-access-zkgzv" pod "170ffbae-db1d-4d75-bc42-6548ea5c74bc" (UID: "170ffbae-db1d-4d75-bc42-6548ea5c74bc"). InnerVolumeSpecName "kube-api-access-zkgzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:00.302209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:00.302182 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/170ffbae-db1d-4d75-bc42-6548ea5c74bc-maas-api-xks-ca" pod "170ffbae-db1d-4d75-bc42-6548ea5c74bc" (UID: "170ffbae-db1d-4d75-bc42-6548ea5c74bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:00.399532 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:00.399441 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkgzv\" (UniqueName: \"kubernetes.io/projected/170ffbae-db1d-4d75-bc42-6548ea5c74bc-kube-api-access-zkgzv\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:00.399532 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:00.399478 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/170ffbae-db1d-4d75-bc42-6548ea5c74bc-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:00.399532 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:00.399488 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/170ffbae-db1d-4d75-bc42-6548ea5c74bc-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:00.728653 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:00.728612 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:56:01.015964 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.015890 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57902142-df3e-4114-ac6c-7d42a300f2f9" path="/var/lib/kubelet/pods/57902142-df3e-4114-ac6c-7d42a300f2f9/volumes" Oct 30 21:56:01.076333 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.076304 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng" Oct 30 21:56:01.076557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.076278 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng" event={"ID":"170ffbae-db1d-4d75-bc42-6548ea5c74bc","Type":"ContainerDied","Data":"a3225f836892b85b640825fe9221577a891afaa19bf717ced981c96739ae9b6a"} Oct 30 21:56:01.076557 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.076522 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3225f836892b85b640825fe9221577a891afaa19bf717ced981c96739ae9b6a" Oct 30 21:56:01.747120 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.747094 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:56:01.824373 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.824338 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-data\") pod \"44fcef78-1e16-4d54-9306-52f006c0bb9f\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " Oct 30 21:56:01.824565 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.824382 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-token\" (UniqueName: \"kubernetes.io/projected/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-token\") pod \"44fcef78-1e16-4d54-9306-52f006c0bb9f\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " Oct 30 21:56:01.824565 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.824403 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-workload-socket\") pod \"44fcef78-1e16-4d54-9306-52f006c0bb9f\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " Oct 30 21:56:01.824565 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.824439 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-podinfo\") pod \"44fcef78-1e16-4d54-9306-52f006c0bb9f\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " Oct 30 21:56:01.824814 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.824666 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-data" pod "44fcef78-1e16-4d54-9306-52f006c0bb9f" (UID: "44fcef78-1e16-4d54-9306-52f006c0bb9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:01.824814 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.824766 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-credential-socket\") pod \"44fcef78-1e16-4d54-9306-52f006c0bb9f\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " Oct 30 21:56:01.824814 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.824793 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-workload-socket" pod "44fcef78-1e16-4d54-9306-52f006c0bb9f" (UID: "44fcef78-1e16-4d54-9306-52f006c0bb9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:01.824985 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.824967 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-envoy\") pod \"44fcef78-1e16-4d54-9306-52f006c0bb9f\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " Oct 30 21:56:01.825067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.825007 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-credential-socket" pod "44fcef78-1e16-4d54-9306-52f006c0bb9f" (UID: "44fcef78-1e16-4d54-9306-52f006c0bb9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:01.825067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.825013 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44fcef78-1e16-4d54-9306-52f006c0bb9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44fcef78-1e16-4d54-9306-52f006c0bb9f-istiod-ca-cert\") pod \"44fcef78-1e16-4d54-9306-52f006c0bb9f\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " Oct 30 21:56:01.825195 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.825133 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-workload-certs\") pod \"44fcef78-1e16-4d54-9306-52f006c0bb9f\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " Oct 30 21:56:01.825277 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.825257 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44fcef78-1e16-4d54-9306-52f006c0bb9f-kube-api-access-d77cw\" (UniqueName: \"kubernetes.io/projected/44fcef78-1e16-4d54-9306-52f006c0bb9f-kube-api-access-d77cw\") pod \"44fcef78-1e16-4d54-9306-52f006c0bb9f\" (UID: \"44fcef78-1e16-4d54-9306-52f006c0bb9f\") " Oct 30 21:56:01.825426 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.825403 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44fcef78-1e16-4d54-9306-52f006c0bb9f-istiod-ca-cert" pod "44fcef78-1e16-4d54-9306-52f006c0bb9f" (UID: "44fcef78-1e16-4d54-9306-52f006c0bb9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:01.825978 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.825805 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:01.825978 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.825830 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:01.825978 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.825844 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:01.825978 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.825858 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44fcef78-1e16-4d54-9306-52f006c0bb9f-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:01.827738 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.826137 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-workload-certs" pod "44fcef78-1e16-4d54-9306-52f006c0bb9f" (UID: "44fcef78-1e16-4d54-9306-52f006c0bb9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:01.827870 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.827847 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-token" pod "44fcef78-1e16-4d54-9306-52f006c0bb9f" (UID: "44fcef78-1e16-4d54-9306-52f006c0bb9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:01.836983 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.836955 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-envoy" pod "44fcef78-1e16-4d54-9306-52f006c0bb9f" (UID: "44fcef78-1e16-4d54-9306-52f006c0bb9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:01.837123 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.837005 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44fcef78-1e16-4d54-9306-52f006c0bb9f-kube-api-access-d77cw" pod "44fcef78-1e16-4d54-9306-52f006c0bb9f" (UID: "44fcef78-1e16-4d54-9306-52f006c0bb9f"). InnerVolumeSpecName "kube-api-access-d77cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:01.837347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.837328 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-podinfo" pod "44fcef78-1e16-4d54-9306-52f006c0bb9f" (UID: "44fcef78-1e16-4d54-9306-52f006c0bb9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:56:01.926915 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.926871 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d77cw\" (UniqueName: \"kubernetes.io/projected/44fcef78-1e16-4d54-9306-52f006c0bb9f-kube-api-access-d77cw\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:01.926915 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.926912 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:01.926915 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.926926 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:01.927211 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.926940 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:01.927211 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:01.926953 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44fcef78-1e16-4d54-9306-52f006c0bb9f-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:02.086167 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:02.086063 4926 generic.go:356] "Generic (PLEG): container finished" podID="44fcef78-1e16-4d54-9306-52f006c0bb9f" containerID="0e29aabd94a498c95d50dd7e7fc44bb7fa88cf6d648357afaf19538380a24582" exitCode=0 Oct 30 21:56:02.086391 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:02.086354 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" event={"ID":"44fcef78-1e16-4d54-9306-52f006c0bb9f","Type":"ContainerDied","Data":"0e29aabd94a498c95d50dd7e7fc44bb7fa88cf6d648357afaf19538380a24582"} Oct 30 21:56:02.086516 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:02.086408 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" Oct 30 21:56:02.086516 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:02.086427 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5" event={"ID":"44fcef78-1e16-4d54-9306-52f006c0bb9f","Type":"ContainerDied","Data":"f26d782a4a27587dd2d4edf618dd10cb79ebf977e3469e825d1ecc252541338b"} Oct 30 21:56:02.086516 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:02.086451 4926 scope.go:122] "RemoveContainer" containerID="0e29aabd94a498c95d50dd7e7fc44bb7fa88cf6d648357afaf19538380a24582" Oct 30 21:56:02.098805 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:02.098785 4926 scope.go:122] "RemoveContainer" containerID="078e4011583f679da1086c5dbc33fd6e4e37f7bd60d39bc2f56dd5e2ed56107c" Oct 30 21:56:02.109597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:02.109580 4926 scope.go:122] "RemoveContainer" containerID="0e29aabd94a498c95d50dd7e7fc44bb7fa88cf6d648357afaf19538380a24582" Oct 30 21:56:02.109885 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:56:02.109865 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e29aabd94a498c95d50dd7e7fc44bb7fa88cf6d648357afaf19538380a24582\": container with ID starting with 0e29aabd94a498c95d50dd7e7fc44bb7fa88cf6d648357afaf19538380a24582 not found: ID does not exist" containerID="0e29aabd94a498c95d50dd7e7fc44bb7fa88cf6d648357afaf19538380a24582" Oct 30 21:56:02.109939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:02.109896 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e29aabd94a498c95d50dd7e7fc44bb7fa88cf6d648357afaf19538380a24582"} err="failed to get container status \"0e29aabd94a498c95d50dd7e7fc44bb7fa88cf6d648357afaf19538380a24582\": rpc error: code = NotFound desc = could not find container \"0e29aabd94a498c95d50dd7e7fc44bb7fa88cf6d648357afaf19538380a24582\": container with ID starting with 0e29aabd94a498c95d50dd7e7fc44bb7fa88cf6d648357afaf19538380a24582 not found: ID does not exist" Oct 30 21:56:02.109939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:02.109915 4926 scope.go:122] "RemoveContainer" containerID="078e4011583f679da1086c5dbc33fd6e4e37f7bd60d39bc2f56dd5e2ed56107c" Oct 30 21:56:02.110203 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:56:02.110187 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"078e4011583f679da1086c5dbc33fd6e4e37f7bd60d39bc2f56dd5e2ed56107c\": container with ID starting with 078e4011583f679da1086c5dbc33fd6e4e37f7bd60d39bc2f56dd5e2ed56107c not found: ID does not exist" containerID="078e4011583f679da1086c5dbc33fd6e4e37f7bd60d39bc2f56dd5e2ed56107c" Oct 30 21:56:02.110252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:02.110210 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"078e4011583f679da1086c5dbc33fd6e4e37f7bd60d39bc2f56dd5e2ed56107c"} err="failed to get container status \"078e4011583f679da1086c5dbc33fd6e4e37f7bd60d39bc2f56dd5e2ed56107c\": rpc error: code = NotFound desc = could not find container \"078e4011583f679da1086c5dbc33fd6e4e37f7bd60d39bc2f56dd5e2ed56107c\": container with ID starting with 078e4011583f679da1086c5dbc33fd6e4e37f7bd60d39bc2f56dd5e2ed56107c not found: ID does not exist" Oct 30 21:56:02.164404 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:02.164359 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5"] Oct 30 21:56:02.173641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:02.173601 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9cd83f-openshift-default-776599bcdc-fdst5"] Oct 30 21:56:03.022402 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:03.022362 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44fcef78-1e16-4d54-9306-52f006c0bb9f" path="/var/lib/kubelet/pods/44fcef78-1e16-4d54-9306-52f006c0bb9f/volumes" Oct 30 21:56:04.699320 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:04.699274 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj"] Oct 30 21:56:04.699951 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:04.699537 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" podUID="3bf71b99-a679-4d29-b081-5d26c7abfcde" containerName="istio-proxy" containerID="cri-o://0580a012b0114b0f9f738092a9f4996c1f47990f18404e33143fb6d81782a20e" gracePeriod=30 Oct 30 21:56:04.834082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:04.828929 4926 patch_prober.go:29] interesting pod/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:56:04.834082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:04.828991 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" podUID="3bf71b99-a679-4d29-b081-5d26c7abfcde" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:56:05.504297 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.504252 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd"] Oct 30 21:56:05.505042 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.505009 4926 cpu_manager.go:414] "containerMap: removing container" podUID="57902142-df3e-4114-ac6c-7d42a300f2f9" containerName="payload-processing" Oct 30 21:56:05.505102 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.505049 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57902142-df3e-4114-ac6c-7d42a300f2f9" containerName="payload-processing" Oct 30 21:56:05.505140 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.505100 4926 cpu_manager.go:414] "containerMap: removing container" podUID="44fcef78-1e16-4d54-9306-52f006c0bb9f" containerName="istio-proxy" Oct 30 21:56:05.505140 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.505109 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44fcef78-1e16-4d54-9306-52f006c0bb9f" containerName="istio-proxy" Oct 30 21:56:05.505140 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.505120 4926 cpu_manager.go:414] "containerMap: removing container" podUID="170ffbae-db1d-4d75-bc42-6548ea5c74bc" containerName="revoke-keys" Oct 30 21:56:05.505140 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.505129 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="170ffbae-db1d-4d75-bc42-6548ea5c74bc" containerName="revoke-keys" Oct 30 21:56:05.505274 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.505141 4926 cpu_manager.go:414] "containerMap: removing container" podUID="44fcef78-1e16-4d54-9306-52f006c0bb9f" containerName="istio-proxy" Oct 30 21:56:05.505274 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.505150 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44fcef78-1e16-4d54-9306-52f006c0bb9f" containerName="istio-proxy" Oct 30 21:56:05.505342 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.505315 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="44fcef78-1e16-4d54-9306-52f006c0bb9f" containerName="istio-proxy" Oct 30 21:56:05.505342 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.505328 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="170ffbae-db1d-4d75-bc42-6548ea5c74bc" containerName="revoke-keys" Oct 30 21:56:05.505342 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.505339 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="44fcef78-1e16-4d54-9306-52f006c0bb9f" containerName="istio-proxy" Oct 30 21:56:05.505437 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.505357 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="57902142-df3e-4114-ac6c-7d42a300f2f9" containerName="payload-processing" Oct 30 21:56:05.511307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.511281 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd" Oct 30 21:56:05.540307 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.540271 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:56:05.540480 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.540326 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:56:05.540480 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.540348 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:56:05.540480 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.540385 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:56:05.568021 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.567988 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd"] Oct 30 21:56:05.572316 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.572279 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e075f3-e59c-4bbd-85af-311b46703f80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd\" (UID: \"29e075f3-e59c-4bbd-85af-311b46703f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd" Oct 30 21:56:05.572500 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.572342 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e075f3-e59c-4bbd-85af-311b46703f80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd\" (UID: \"29e075f3-e59c-4bbd-85af-311b46703f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd" Oct 30 21:56:05.572632 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.572606 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5rpw\" (UniqueName: \"kubernetes.io/projected/29e075f3-e59c-4bbd-85af-311b46703f80-kube-api-access-k5rpw\") pod \"maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd\" (UID: \"29e075f3-e59c-4bbd-85af-311b46703f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd" Oct 30 21:56:05.674287 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.674246 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e075f3-e59c-4bbd-85af-311b46703f80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd\" (UID: \"29e075f3-e59c-4bbd-85af-311b46703f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd" Oct 30 21:56:05.674508 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.674381 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5rpw\" (UniqueName: \"kubernetes.io/projected/29e075f3-e59c-4bbd-85af-311b46703f80-kube-api-access-k5rpw\") pod \"maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd\" (UID: \"29e075f3-e59c-4bbd-85af-311b46703f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd" Oct 30 21:56:05.674508 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.674466 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e075f3-e59c-4bbd-85af-311b46703f80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd\" (UID: \"29e075f3-e59c-4bbd-85af-311b46703f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd" Oct 30 21:56:05.675134 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.675104 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e075f3-e59c-4bbd-85af-311b46703f80-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd\" (UID: \"29e075f3-e59c-4bbd-85af-311b46703f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd" Oct 30 21:56:05.676907 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.676881 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e075f3-e59c-4bbd-85af-311b46703f80-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd\" (UID: \"29e075f3-e59c-4bbd-85af-311b46703f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd" Oct 30 21:56:05.702788 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.702752 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5rpw\" (UniqueName: \"kubernetes.io/projected/29e075f3-e59c-4bbd-85af-311b46703f80-kube-api-access-k5rpw\") pod \"maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd\" (UID: \"29e075f3-e59c-4bbd-85af-311b46703f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd" Oct 30 21:56:05.827140 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:05.827021 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd" Oct 30 21:56:06.142407 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:06.142375 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd"] Oct 30 21:56:06.143120 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:56:06.143087 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e075f3_e59c_4bbd_85af_311b46703f80.slice/crio-eb2d1c1031599d0f718f8f7e8df7ba075c790216a1b0aa9f53e2b7bfdc2b8cc2 WatchSource:0}: Error finding container eb2d1c1031599d0f718f8f7e8df7ba075c790216a1b0aa9f53e2b7bfdc2b8cc2: Status 404 returned error can't find the container with id eb2d1c1031599d0f718f8f7e8df7ba075c790216a1b0aa9f53e2b7bfdc2b8cc2 Oct 30 21:56:07.133987 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:07.133947 4926 generic.go:356] "Generic (PLEG): container finished" podID="29e075f3-e59c-4bbd-85af-311b46703f80" containerID="3f3cf194fdc0f8926ec908e5aabfba0dccaa36174e75d05186d303026fbd3744" exitCode=0 Oct 30 21:56:07.134458 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:07.134049 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd" event={"ID":"29e075f3-e59c-4bbd-85af-311b46703f80","Type":"ContainerDied","Data":"3f3cf194fdc0f8926ec908e5aabfba0dccaa36174e75d05186d303026fbd3744"} Oct 30 21:56:07.134458 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:07.134114 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd" event={"ID":"29e075f3-e59c-4bbd-85af-311b46703f80","Type":"ContainerStarted","Data":"eb2d1c1031599d0f718f8f7e8df7ba075c790216a1b0aa9f53e2b7bfdc2b8cc2"} Oct 30 21:56:07.486171 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:56:07.486121 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:56:08.334000 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:08.333970 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd" Oct 30 21:56:08.406963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:08.406870 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29e075f3-e59c-4bbd-85af-311b46703f80-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e075f3-e59c-4bbd-85af-311b46703f80-maas-api-xks-ca\") pod \"29e075f3-e59c-4bbd-85af-311b46703f80\" (UID: \"29e075f3-e59c-4bbd-85af-311b46703f80\") " Oct 30 21:56:08.407256 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:08.407003 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29e075f3-e59c-4bbd-85af-311b46703f80-kube-api-access-k5rpw\" (UniqueName: \"kubernetes.io/projected/29e075f3-e59c-4bbd-85af-311b46703f80-kube-api-access-k5rpw\") pod \"29e075f3-e59c-4bbd-85af-311b46703f80\" (UID: \"29e075f3-e59c-4bbd-85af-311b46703f80\") " Oct 30 21:56:08.407256 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:08.407073 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29e075f3-e59c-4bbd-85af-311b46703f80-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e075f3-e59c-4bbd-85af-311b46703f80-maas-api-openshift-ca\") pod \"29e075f3-e59c-4bbd-85af-311b46703f80\" (UID: \"29e075f3-e59c-4bbd-85af-311b46703f80\") " Oct 30 21:56:08.407886 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:08.407859 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e075f3-e59c-4bbd-85af-311b46703f80-maas-api-openshift-ca" pod "29e075f3-e59c-4bbd-85af-311b46703f80" (UID: "29e075f3-e59c-4bbd-85af-311b46703f80"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:08.409374 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:08.409349 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e075f3-e59c-4bbd-85af-311b46703f80-maas-api-xks-ca" pod "29e075f3-e59c-4bbd-85af-311b46703f80" (UID: "29e075f3-e59c-4bbd-85af-311b46703f80"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:08.414434 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:08.414394 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e075f3-e59c-4bbd-85af-311b46703f80-kube-api-access-k5rpw" pod "29e075f3-e59c-4bbd-85af-311b46703f80" (UID: "29e075f3-e59c-4bbd-85af-311b46703f80"). InnerVolumeSpecName "kube-api-access-k5rpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:08.509965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:08.509920 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29e075f3-e59c-4bbd-85af-311b46703f80-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:08.509965 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:08.509966 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5rpw\" (UniqueName: \"kubernetes.io/projected/29e075f3-e59c-4bbd-85af-311b46703f80-kube-api-access-k5rpw\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:08.510296 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:08.509982 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29e075f3-e59c-4bbd-85af-311b46703f80-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:09.047724 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:09.047684 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c"] Oct 30 21:56:09.059114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:09.059079 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df9b4e38-8nr8c"] Oct 30 21:56:09.152045 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:09.152000 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd" Oct 30 21:56:09.152245 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:09.152021 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd" event={"ID":"29e075f3-e59c-4bbd-85af-311b46703f80","Type":"ContainerDied","Data":"eb2d1c1031599d0f718f8f7e8df7ba075c790216a1b0aa9f53e2b7bfdc2b8cc2"} Oct 30 21:56:09.152245 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:09.152106 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb2d1c1031599d0f718f8f7e8df7ba075c790216a1b0aa9f53e2b7bfdc2b8cc2" Oct 30 21:56:10.023049 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.023006 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:56:10.134829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.134788 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bf71b99-a679-4d29-b081-5d26c7abfcde-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf71b99-a679-4d29-b081-5d26c7abfcde-istiod-ca-cert\") pod \"3bf71b99-a679-4d29-b081-5d26c7abfcde\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " Oct 30 21:56:10.135088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.134879 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-workload-certs\") pod \"3bf71b99-a679-4d29-b081-5d26c7abfcde\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " Oct 30 21:56:10.135088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.134923 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-envoy\") pod \"3bf71b99-a679-4d29-b081-5d26c7abfcde\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " Oct 30 21:56:10.135227 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.135195 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-token\" (UniqueName: \"kubernetes.io/projected/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-token\") pod \"3bf71b99-a679-4d29-b081-5d26c7abfcde\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " Oct 30 21:56:10.135285 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.135252 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bf71b99-a679-4d29-b081-5d26c7abfcde-istiod-ca-cert" pod "3bf71b99-a679-4d29-b081-5d26c7abfcde" (UID: "3bf71b99-a679-4d29-b081-5d26c7abfcde"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:10.135285 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.135259 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-workload-certs" pod "3bf71b99-a679-4d29-b081-5d26c7abfcde" (UID: "3bf71b99-a679-4d29-b081-5d26c7abfcde"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:10.135285 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.135256 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-podinfo\") pod \"3bf71b99-a679-4d29-b081-5d26c7abfcde\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " Oct 30 21:56:10.135460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.135330 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-credential-socket\") pod \"3bf71b99-a679-4d29-b081-5d26c7abfcde\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " Oct 30 21:56:10.135460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.135364 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bf71b99-a679-4d29-b081-5d26c7abfcde-kube-api-access-tvpz9\" (UniqueName: \"kubernetes.io/projected/3bf71b99-a679-4d29-b081-5d26c7abfcde-kube-api-access-tvpz9\") pod \"3bf71b99-a679-4d29-b081-5d26c7abfcde\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " Oct 30 21:56:10.135460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.135395 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-data\") pod \"3bf71b99-a679-4d29-b081-5d26c7abfcde\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " Oct 30 21:56:10.135460 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.135416 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-workload-socket\") pod \"3bf71b99-a679-4d29-b081-5d26c7abfcde\" (UID: \"3bf71b99-a679-4d29-b081-5d26c7abfcde\") " Oct 30 21:56:10.135814 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.135791 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-credential-socket" pod "3bf71b99-a679-4d29-b081-5d26c7abfcde" (UID: "3bf71b99-a679-4d29-b081-5d26c7abfcde"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:10.135927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.135897 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-data" pod "3bf71b99-a679-4d29-b081-5d26c7abfcde" (UID: "3bf71b99-a679-4d29-b081-5d26c7abfcde"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:10.136114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.136089 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-workload-socket" pod "3bf71b99-a679-4d29-b081-5d26c7abfcde" (UID: "3bf71b99-a679-4d29-b081-5d26c7abfcde"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:10.136396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.136372 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:10.136482 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.136402 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:10.136482 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.136416 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:10.136482 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.136429 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bf71b99-a679-4d29-b081-5d26c7abfcde-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:10.136482 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.136441 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:10.138661 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.138453 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-podinfo" pod "3bf71b99-a679-4d29-b081-5d26c7abfcde" (UID: "3bf71b99-a679-4d29-b081-5d26c7abfcde"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:56:10.138661 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.138519 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-token" pod "3bf71b99-a679-4d29-b081-5d26c7abfcde" (UID: "3bf71b99-a679-4d29-b081-5d26c7abfcde"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:10.138838 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.138680 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-envoy" pod "3bf71b99-a679-4d29-b081-5d26c7abfcde" (UID: "3bf71b99-a679-4d29-b081-5d26c7abfcde"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:10.138838 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.138764 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf71b99-a679-4d29-b081-5d26c7abfcde-kube-api-access-tvpz9" pod "3bf71b99-a679-4d29-b081-5d26c7abfcde" (UID: "3bf71b99-a679-4d29-b081-5d26c7abfcde"). InnerVolumeSpecName "kube-api-access-tvpz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:10.217063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.215337 4926 generic.go:356] "Generic (PLEG): container finished" podID="3bf71b99-a679-4d29-b081-5d26c7abfcde" containerID="0580a012b0114b0f9f738092a9f4996c1f47990f18404e33143fb6d81782a20e" exitCode=0 Oct 30 21:56:10.217063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.215534 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" event={"ID":"3bf71b99-a679-4d29-b081-5d26c7abfcde","Type":"ContainerDied","Data":"0580a012b0114b0f9f738092a9f4996c1f47990f18404e33143fb6d81782a20e"} Oct 30 21:56:10.217063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.215588 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" event={"ID":"3bf71b99-a679-4d29-b081-5d26c7abfcde","Type":"ContainerDied","Data":"555e28684bb3402e83db47849552d7358ca6d7b7f4fa5cda1ed63421563462f6"} Oct 30 21:56:10.217063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.215617 4926 scope.go:122] "RemoveContainer" containerID="0580a012b0114b0f9f738092a9f4996c1f47990f18404e33143fb6d81782a20e" Oct 30 21:56:10.217063 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.215797 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj" Oct 30 21:56:10.229376 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.229333 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2"] Oct 30 21:56:10.236855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.236345 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg"] Oct 30 21:56:10.237528 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.237332 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvpz9\" (UniqueName: \"kubernetes.io/projected/3bf71b99-a679-4d29-b081-5d26c7abfcde-kube-api-access-tvpz9\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:10.237528 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.237367 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:10.237528 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.237385 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:10.237528 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.237398 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bf71b99-a679-4d29-b081-5d26c7abfcde-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:10.246122 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.245072 4926 scope.go:122] "RemoveContainer" containerID="0580a012b0114b0f9f738092a9f4996c1f47990f18404e33143fb6d81782a20e" Oct 30 21:56:10.248711 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:56:10.248620 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0580a012b0114b0f9f738092a9f4996c1f47990f18404e33143fb6d81782a20e\": container with ID starting with 0580a012b0114b0f9f738092a9f4996c1f47990f18404e33143fb6d81782a20e not found: ID does not exist" containerID="0580a012b0114b0f9f738092a9f4996c1f47990f18404e33143fb6d81782a20e" Oct 30 21:56:10.248711 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.248676 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0580a012b0114b0f9f738092a9f4996c1f47990f18404e33143fb6d81782a20e"} err="failed to get container status \"0580a012b0114b0f9f738092a9f4996c1f47990f18404e33143fb6d81782a20e\": rpc error: code = NotFound desc = could not find container \"0580a012b0114b0f9f738092a9f4996c1f47990f18404e33143fb6d81782a20e\": container with ID starting with 0580a012b0114b0f9f738092a9f4996c1f47990f18404e33143fb6d81782a20e not found: ID does not exist" Oct 30 21:56:10.251161 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.251134 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws"] Oct 30 21:56:10.294279 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.294224 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj"] Oct 30 21:56:10.298676 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.298638 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-889195-openshift-default-5b79476475-7v2rj"] Oct 30 21:56:10.734854 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.734655 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px"] Oct 30 21:56:10.745053 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:10.741888 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-e11bef-kq7px"] Oct 30 21:56:11.018093 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:11.018006 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bf71b99-a679-4d29-b081-5d26c7abfcde" path="/var/lib/kubelet/pods/3bf71b99-a679-4d29-b081-5d26c7abfcde/volumes" Oct 30 21:56:11.018528 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:11.018513 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ce6dcce-38b4-4a80-bfcf-a891f45644eb" path="/var/lib/kubelet/pods/6ce6dcce-38b4-4a80-bfcf-a891f45644eb/volumes" Oct 30 21:56:11.018940 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:11.018926 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab95c2d5-2782-4264-85bc-61e70bdc7a00" path="/var/lib/kubelet/pods/ab95c2d5-2782-4264-85bc-61e70bdc7a00/volumes" Oct 30 21:56:14.312379 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:14.312340 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb"] Oct 30 21:56:14.323281 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:14.323245 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f8137c-6qxbb"] Oct 30 21:56:15.016282 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:15.016244 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69f0ac4f-81c8-4648-9b24-2d15b8720d41" path="/var/lib/kubelet/pods/69f0ac4f-81c8-4648-9b24-2d15b8720d41/volumes" Oct 30 21:56:17.897028 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:17.896993 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf"] Oct 30 21:56:17.897720 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:17.897695 4926 cpu_manager.go:414] "containerMap: removing container" podUID="29e075f3-e59c-4bbd-85af-311b46703f80" containerName="revoke-keys" Oct 30 21:56:17.897814 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:17.897721 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e075f3-e59c-4bbd-85af-311b46703f80" containerName="revoke-keys" Oct 30 21:56:17.897814 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:17.897773 4926 cpu_manager.go:414] "containerMap: removing container" podUID="3bf71b99-a679-4d29-b081-5d26c7abfcde" containerName="istio-proxy" Oct 30 21:56:17.897814 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:17.897783 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bf71b99-a679-4d29-b081-5d26c7abfcde" containerName="istio-proxy" Oct 30 21:56:17.897975 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:17.897922 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bf71b99-a679-4d29-b081-5d26c7abfcde" containerName="istio-proxy" Oct 30 21:56:17.897975 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:17.897947 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="29e075f3-e59c-4bbd-85af-311b46703f80" containerName="revoke-keys" Oct 30 21:56:17.901347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:17.901323 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf" Oct 30 21:56:17.901504 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:17.901465 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5"] Oct 30 21:56:17.905253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:17.905230 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5" Oct 30 21:56:17.924286 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:17.924251 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5"] Oct 30 21:56:17.924603 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:17.924578 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:56:17.924742 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:17.924641 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:56:17.931151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:17.931081 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:56:17.933713 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:17.933683 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:56:17.936110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:17.936079 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf"] Oct 30 21:56:18.045348 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:18.045305 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/084dee41-d4ad-4a17-844d-34aa27886965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfd8173a-lt8nf\" (UID: \"084dee41-d4ad-4a17-844d-34aa27886965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf" Oct 30 21:56:18.045550 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:18.045373 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fea7423-7d7a-453f-aa15-b552ba9f43fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab02655b-j5zv5\" (UID: \"8fea7423-7d7a-453f-aa15-b552ba9f43fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5" Oct 30 21:56:18.045550 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:18.045511 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7d8p\" (UniqueName: \"kubernetes.io/projected/8fea7423-7d7a-453f-aa15-b552ba9f43fd-kube-api-access-t7d8p\") pod \"maas-api-revoke-sub-ab02655b-j5zv5\" (UID: \"8fea7423-7d7a-453f-aa15-b552ba9f43fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5" Oct 30 21:56:18.045648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:18.045551 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w25t\" (UniqueName: \"kubernetes.io/projected/084dee41-d4ad-4a17-844d-34aa27886965-kube-api-access-8w25t\") pod \"maas-api-revoke-sub-bfd8173a-lt8nf\" (UID: \"084dee41-d4ad-4a17-844d-34aa27886965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf" Oct 30 21:56:18.045648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:18.045585 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/084dee41-d4ad-4a17-844d-34aa27886965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfd8173a-lt8nf\" (UID: \"084dee41-d4ad-4a17-844d-34aa27886965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf" Oct 30 21:56:18.045753 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:18.045735 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fea7423-7d7a-453f-aa15-b552ba9f43fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab02655b-j5zv5\" (UID: \"8fea7423-7d7a-453f-aa15-b552ba9f43fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5" Oct 30 21:56:18.147376 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:18.147272 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fea7423-7d7a-453f-aa15-b552ba9f43fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab02655b-j5zv5\" (UID: \"8fea7423-7d7a-453f-aa15-b552ba9f43fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5" Oct 30 21:56:18.147580 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:18.147461 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7d8p\" (UniqueName: \"kubernetes.io/projected/8fea7423-7d7a-453f-aa15-b552ba9f43fd-kube-api-access-t7d8p\") pod \"maas-api-revoke-sub-ab02655b-j5zv5\" (UID: \"8fea7423-7d7a-453f-aa15-b552ba9f43fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5" Oct 30 21:56:18.147580 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:18.147515 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w25t\" (UniqueName: \"kubernetes.io/projected/084dee41-d4ad-4a17-844d-34aa27886965-kube-api-access-8w25t\") pod \"maas-api-revoke-sub-bfd8173a-lt8nf\" (UID: \"084dee41-d4ad-4a17-844d-34aa27886965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf" Oct 30 21:56:18.147580 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:18.147565 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/084dee41-d4ad-4a17-844d-34aa27886965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfd8173a-lt8nf\" (UID: \"084dee41-d4ad-4a17-844d-34aa27886965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf" Oct 30 21:56:18.147773 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:18.147724 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fea7423-7d7a-453f-aa15-b552ba9f43fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab02655b-j5zv5\" (UID: \"8fea7423-7d7a-453f-aa15-b552ba9f43fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5" Oct 30 21:56:18.147832 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:18.147783 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/084dee41-d4ad-4a17-844d-34aa27886965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfd8173a-lt8nf\" (UID: \"084dee41-d4ad-4a17-844d-34aa27886965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf" Oct 30 21:56:18.148411 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:18.148381 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/084dee41-d4ad-4a17-844d-34aa27886965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfd8173a-lt8nf\" (UID: \"084dee41-d4ad-4a17-844d-34aa27886965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf" Oct 30 21:56:18.148561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:18.148436 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fea7423-7d7a-453f-aa15-b552ba9f43fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab02655b-j5zv5\" (UID: \"8fea7423-7d7a-453f-aa15-b552ba9f43fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5" Oct 30 21:56:18.150104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:18.150074 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fea7423-7d7a-453f-aa15-b552ba9f43fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab02655b-j5zv5\" (UID: \"8fea7423-7d7a-453f-aa15-b552ba9f43fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5" Oct 30 21:56:18.150232 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:18.150176 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/084dee41-d4ad-4a17-844d-34aa27886965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfd8173a-lt8nf\" (UID: \"084dee41-d4ad-4a17-844d-34aa27886965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf" Oct 30 21:56:18.169086 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:18.169025 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7d8p\" (UniqueName: \"kubernetes.io/projected/8fea7423-7d7a-453f-aa15-b552ba9f43fd-kube-api-access-t7d8p\") pod \"maas-api-revoke-sub-ab02655b-j5zv5\" (UID: \"8fea7423-7d7a-453f-aa15-b552ba9f43fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5" Oct 30 21:56:18.177324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:18.177296 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w25t\" (UniqueName: \"kubernetes.io/projected/084dee41-d4ad-4a17-844d-34aa27886965-kube-api-access-8w25t\") pod \"maas-api-revoke-sub-bfd8173a-lt8nf\" (UID: \"084dee41-d4ad-4a17-844d-34aa27886965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf" Oct 30 21:56:18.217154 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:18.217113 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf" Oct 30 21:56:18.229408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:18.229382 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5" Oct 30 21:56:18.545108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:18.545076 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf"] Oct 30 21:56:18.548224 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:56:18.548192 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod084dee41_d4ad_4a17_844d_34aa27886965.slice/crio-cb47061229106ca053b2c71c23a9f195925ab819b496da1675fc90cc868068d4 WatchSource:0}: Error finding container cb47061229106ca053b2c71c23a9f195925ab819b496da1675fc90cc868068d4: Status 404 returned error can't find the container with id cb47061229106ca053b2c71c23a9f195925ab819b496da1675fc90cc868068d4 Oct 30 21:56:18.592980 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:18.592942 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5"] Oct 30 21:56:18.593706 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:56:18.593680 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fea7423_7d7a_453f_aa15_b552ba9f43fd.slice/crio-ecb65f015b383d0b57f448de28b9364e2dc9a0573d4153435231d0ee2fa29e74 WatchSource:0}: Error finding container ecb65f015b383d0b57f448de28b9364e2dc9a0573d4153435231d0ee2fa29e74: Status 404 returned error can't find the container with id ecb65f015b383d0b57f448de28b9364e2dc9a0573d4153435231d0ee2fa29e74 Oct 30 21:56:19.059766 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:19.059719 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg"] Oct 30 21:56:19.066521 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:19.066483 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-baf22847-xp5bg"] Oct 30 21:56:19.079313 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:19.079278 4926 scope.go:122] "RemoveContainer" containerID="6c1de8c136d42d1108109c3b34a24874649c45cc4d04d697add81ce42d91c415" Oct 30 21:56:19.091571 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:19.091540 4926 scope.go:122] "RemoveContainer" containerID="1e38251da8d3bf3179c15870c641c964f0f5ffb5751501718f8d9e25cd939d2d" Oct 30 21:56:19.102803 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:19.102776 4926 scope.go:122] "RemoveContainer" containerID="0a5975c8ab3d41333e71ac7cef1acef508b766b18c71d25db29282a4d785a39d" Oct 30 21:56:19.340674 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:19.340624 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf" event={"ID":"084dee41-d4ad-4a17-844d-34aa27886965","Type":"ContainerStarted","Data":"66f9d5f1ad029a8ea28bf61da3ad2dadbeb7c95c84ad5a794db0df4136da576f"} Oct 30 21:56:19.340885 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:19.340722 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf" event={"ID":"084dee41-d4ad-4a17-844d-34aa27886965","Type":"ContainerStarted","Data":"cb47061229106ca053b2c71c23a9f195925ab819b496da1675fc90cc868068d4"} Oct 30 21:56:19.342534 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:19.342500 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5" event={"ID":"8fea7423-7d7a-453f-aa15-b552ba9f43fd","Type":"ContainerStarted","Data":"c2c3d173253d43de317b4d7a7b9b91a19c653f7e16c4a393401a929a717ec8e2"} Oct 30 21:56:19.342685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:19.342553 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5" event={"ID":"8fea7423-7d7a-453f-aa15-b552ba9f43fd","Type":"ContainerStarted","Data":"ecb65f015b383d0b57f448de28b9364e2dc9a0573d4153435231d0ee2fa29e74"} Oct 30 21:56:19.385230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:19.385126 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf" podStartSLOduration=2.385110594 podStartE2EDuration="2.385110594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:19.38361914 +0000 UTC m=+3366.504692887" watchObservedRunningTime="2026-10-30 21:56:19.385110594 +0000 UTC m=+3366.506184339" Oct 30 21:56:19.429390 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:19.429320 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5" podStartSLOduration=2.429299049 podStartE2EDuration="2.429299049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:19.42635036 +0000 UTC m=+3366.547424106" watchObservedRunningTime="2026-10-30 21:56:19.429299049 +0000 UTC m=+3366.550372796" Oct 30 21:56:20.353481 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:20.353442 4926 generic.go:356] "Generic (PLEG): container finished" podID="084dee41-d4ad-4a17-844d-34aa27886965" containerID="66f9d5f1ad029a8ea28bf61da3ad2dadbeb7c95c84ad5a794db0df4136da576f" exitCode=0 Oct 30 21:56:20.353931 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:20.353525 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf" event={"ID":"084dee41-d4ad-4a17-844d-34aa27886965","Type":"ContainerDied","Data":"66f9d5f1ad029a8ea28bf61da3ad2dadbeb7c95c84ad5a794db0df4136da576f"} Oct 30 21:56:20.355439 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:20.355412 4926 generic.go:356] "Generic (PLEG): container finished" podID="8fea7423-7d7a-453f-aa15-b552ba9f43fd" containerID="c2c3d173253d43de317b4d7a7b9b91a19c653f7e16c4a393401a929a717ec8e2" exitCode=0 Oct 30 21:56:20.355578 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:20.355438 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5" event={"ID":"8fea7423-7d7a-453f-aa15-b552ba9f43fd","Type":"ContainerDied","Data":"c2c3d173253d43de317b4d7a7b9b91a19c653f7e16c4a393401a929a717ec8e2"} Oct 30 21:56:21.019234 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.019198 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1223d4a3-7b03-487b-a2d4-f9856f89d6a7" path="/var/lib/kubelet/pods/1223d4a3-7b03-487b-a2d4-f9856f89d6a7/volumes" Oct 30 21:56:21.603616 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.603575 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5" Oct 30 21:56:21.626425 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.626364 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf" Oct 30 21:56:21.654655 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.654615 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fea7423-7d7a-453f-aa15-b552ba9f43fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fea7423-7d7a-453f-aa15-b552ba9f43fd-maas-api-openshift-ca\") pod \"8fea7423-7d7a-453f-aa15-b552ba9f43fd\" (UID: \"8fea7423-7d7a-453f-aa15-b552ba9f43fd\") " Oct 30 21:56:21.654865 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.654673 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/084dee41-d4ad-4a17-844d-34aa27886965-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/084dee41-d4ad-4a17-844d-34aa27886965-maas-api-openshift-ca\") pod \"084dee41-d4ad-4a17-844d-34aa27886965\" (UID: \"084dee41-d4ad-4a17-844d-34aa27886965\") " Oct 30 21:56:21.654865 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.654721 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/084dee41-d4ad-4a17-844d-34aa27886965-kube-api-access-8w25t\" (UniqueName: \"kubernetes.io/projected/084dee41-d4ad-4a17-844d-34aa27886965-kube-api-access-8w25t\") pod \"084dee41-d4ad-4a17-844d-34aa27886965\" (UID: \"084dee41-d4ad-4a17-844d-34aa27886965\") " Oct 30 21:56:21.654865 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.654777 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/084dee41-d4ad-4a17-844d-34aa27886965-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/084dee41-d4ad-4a17-844d-34aa27886965-maas-api-xks-ca\") pod \"084dee41-d4ad-4a17-844d-34aa27886965\" (UID: \"084dee41-d4ad-4a17-844d-34aa27886965\") " Oct 30 21:56:21.655060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.654913 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fea7423-7d7a-453f-aa15-b552ba9f43fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fea7423-7d7a-453f-aa15-b552ba9f43fd-maas-api-xks-ca\") pod \"8fea7423-7d7a-453f-aa15-b552ba9f43fd\" (UID: \"8fea7423-7d7a-453f-aa15-b552ba9f43fd\") " Oct 30 21:56:21.655060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.654950 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fea7423-7d7a-453f-aa15-b552ba9f43fd-kube-api-access-t7d8p\" (UniqueName: \"kubernetes.io/projected/8fea7423-7d7a-453f-aa15-b552ba9f43fd-kube-api-access-t7d8p\") pod \"8fea7423-7d7a-453f-aa15-b552ba9f43fd\" (UID: \"8fea7423-7d7a-453f-aa15-b552ba9f43fd\") " Oct 30 21:56:21.655188 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.655080 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fea7423-7d7a-453f-aa15-b552ba9f43fd-maas-api-openshift-ca" pod "8fea7423-7d7a-453f-aa15-b552ba9f43fd" (UID: "8fea7423-7d7a-453f-aa15-b552ba9f43fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:21.655188 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.655127 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/084dee41-d4ad-4a17-844d-34aa27886965-maas-api-openshift-ca" pod "084dee41-d4ad-4a17-844d-34aa27886965" (UID: "084dee41-d4ad-4a17-844d-34aa27886965"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:21.655417 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.655395 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fea7423-7d7a-453f-aa15-b552ba9f43fd-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:21.655478 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.655423 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/084dee41-d4ad-4a17-844d-34aa27886965-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:21.657328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.657299 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fea7423-7d7a-453f-aa15-b552ba9f43fd-kube-api-access-t7d8p" pod "8fea7423-7d7a-453f-aa15-b552ba9f43fd" (UID: "8fea7423-7d7a-453f-aa15-b552ba9f43fd"). InnerVolumeSpecName "kube-api-access-t7d8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:21.657444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.657299 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fea7423-7d7a-453f-aa15-b552ba9f43fd-maas-api-xks-ca" pod "8fea7423-7d7a-453f-aa15-b552ba9f43fd" (UID: "8fea7423-7d7a-453f-aa15-b552ba9f43fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:21.657444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.657302 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/084dee41-d4ad-4a17-844d-34aa27886965-maas-api-xks-ca" pod "084dee41-d4ad-4a17-844d-34aa27886965" (UID: "084dee41-d4ad-4a17-844d-34aa27886965"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:21.657444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.657405 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/084dee41-d4ad-4a17-844d-34aa27886965-kube-api-access-8w25t" pod "084dee41-d4ad-4a17-844d-34aa27886965" (UID: "084dee41-d4ad-4a17-844d-34aa27886965"). InnerVolumeSpecName "kube-api-access-8w25t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:21.756001 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.755969 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7d8p\" (UniqueName: \"kubernetes.io/projected/8fea7423-7d7a-453f-aa15-b552ba9f43fd-kube-api-access-t7d8p\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:21.756001 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.755998 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w25t\" (UniqueName: \"kubernetes.io/projected/084dee41-d4ad-4a17-844d-34aa27886965-kube-api-access-8w25t\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:21.756001 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.756010 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/084dee41-d4ad-4a17-844d-34aa27886965-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:21.756302 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.756019 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fea7423-7d7a-453f-aa15-b552ba9f43fd-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:21.957834 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.957785 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9"] Oct 30 21:56:21.958607 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.958582 4926 cpu_manager.go:414] "containerMap: removing container" podUID="8fea7423-7d7a-453f-aa15-b552ba9f43fd" containerName="revoke-keys" Oct 30 21:56:21.958658 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.958612 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fea7423-7d7a-453f-aa15-b552ba9f43fd" containerName="revoke-keys" Oct 30 21:56:21.958702 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.958666 4926 cpu_manager.go:414] "containerMap: removing container" podUID="084dee41-d4ad-4a17-844d-34aa27886965" containerName="revoke-keys" Oct 30 21:56:21.958702 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.958678 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="084dee41-d4ad-4a17-844d-34aa27886965" containerName="revoke-keys" Oct 30 21:56:21.958882 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.958866 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="084dee41-d4ad-4a17-844d-34aa27886965" containerName="revoke-keys" Oct 30 21:56:21.958919 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.958896 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fea7423-7d7a-453f-aa15-b552ba9f43fd" containerName="revoke-keys" Oct 30 21:56:21.963299 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.963273 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9" Oct 30 21:56:21.980224 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:21.980196 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9"] Oct 30 21:56:22.058026 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:22.057979 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e1f5fe-1324-4394-956d-8e315c4c3151-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9\" (UID: \"94e1f5fe-1324-4394-956d-8e315c4c3151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9" Oct 30 21:56:22.058203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:22.058175 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e1f5fe-1324-4394-956d-8e315c4c3151-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9\" (UID: \"94e1f5fe-1324-4394-956d-8e315c4c3151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9" Oct 30 21:56:22.058266 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:22.058254 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wblj\" (UniqueName: \"kubernetes.io/projected/94e1f5fe-1324-4394-956d-8e315c4c3151-kube-api-access-9wblj\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9\" (UID: \"94e1f5fe-1324-4394-956d-8e315c4c3151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9" Oct 30 21:56:22.159487 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:22.159439 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e1f5fe-1324-4394-956d-8e315c4c3151-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9\" (UID: \"94e1f5fe-1324-4394-956d-8e315c4c3151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9" Oct 30 21:56:22.159709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:22.159527 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e1f5fe-1324-4394-956d-8e315c4c3151-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9\" (UID: \"94e1f5fe-1324-4394-956d-8e315c4c3151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9" Oct 30 21:56:22.159709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:22.159585 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wblj\" (UniqueName: \"kubernetes.io/projected/94e1f5fe-1324-4394-956d-8e315c4c3151-kube-api-access-9wblj\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9\" (UID: \"94e1f5fe-1324-4394-956d-8e315c4c3151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9" Oct 30 21:56:22.160297 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:22.160270 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e1f5fe-1324-4394-956d-8e315c4c3151-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9\" (UID: \"94e1f5fe-1324-4394-956d-8e315c4c3151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9" Oct 30 21:56:22.162488 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:22.162463 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e1f5fe-1324-4394-956d-8e315c4c3151-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9\" (UID: \"94e1f5fe-1324-4394-956d-8e315c4c3151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9" Oct 30 21:56:22.181840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:22.181812 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wblj\" (UniqueName: \"kubernetes.io/projected/94e1f5fe-1324-4394-956d-8e315c4c3151-kube-api-access-9wblj\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9\" (UID: \"94e1f5fe-1324-4394-956d-8e315c4c3151\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9" Oct 30 21:56:22.279926 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:22.279838 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9" Oct 30 21:56:22.388583 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:22.388549 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf" Oct 30 21:56:22.389224 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:22.389187 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf" event={"ID":"084dee41-d4ad-4a17-844d-34aa27886965","Type":"ContainerDied","Data":"cb47061229106ca053b2c71c23a9f195925ab819b496da1675fc90cc868068d4"} Oct 30 21:56:22.389352 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:22.389251 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb47061229106ca053b2c71c23a9f195925ab819b496da1675fc90cc868068d4" Oct 30 21:56:22.391015 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:22.390987 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5" event={"ID":"8fea7423-7d7a-453f-aa15-b552ba9f43fd","Type":"ContainerDied","Data":"ecb65f015b383d0b57f448de28b9364e2dc9a0573d4153435231d0ee2fa29e74"} Oct 30 21:56:22.391133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:22.391062 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecb65f015b383d0b57f448de28b9364e2dc9a0573d4153435231d0ee2fa29e74" Oct 30 21:56:22.391133 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:22.391128 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5" Oct 30 21:56:22.588770 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:22.588736 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9"] Oct 30 21:56:22.589760 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:56:22.589737 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e1f5fe_1324_4394_956d_8e315c4c3151.slice/crio-d37d3bae05e002b94441d3f1b87c1b871a600670e10de739b04ba4db0c9112a0 WatchSource:0}: Error finding container d37d3bae05e002b94441d3f1b87c1b871a600670e10de739b04ba4db0c9112a0: Status 404 returned error can't find the container with id d37d3bae05e002b94441d3f1b87c1b871a600670e10de739b04ba4db0c9112a0 Oct 30 21:56:23.400699 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:23.400642 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9" event={"ID":"94e1f5fe-1324-4394-956d-8e315c4c3151","Type":"ContainerStarted","Data":"716c7df33597e2a5c22c23f352f087f5512cdc4c558154a64d0057b1f48a5389"} Oct 30 21:56:23.401145 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:23.400748 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9" event={"ID":"94e1f5fe-1324-4394-956d-8e315c4c3151","Type":"ContainerStarted","Data":"d37d3bae05e002b94441d3f1b87c1b871a600670e10de739b04ba4db0c9112a0"} Oct 30 21:56:23.459426 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:23.459370 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9" podStartSLOduration=2.459354933 podStartE2EDuration="2.459354933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:23.455328755 +0000 UTC m=+3370.576402503" watchObservedRunningTime="2026-10-30 21:56:23.459354933 +0000 UTC m=+3370.580428679" Oct 30 21:56:24.190631 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.190583 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6"] Oct 30 21:56:24.196054 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.196006 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6" Oct 30 21:56:24.222409 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.222368 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6"] Oct 30 21:56:24.232558 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.232528 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh"] Oct 30 21:56:24.236632 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.236614 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh" Oct 30 21:56:24.251787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.251753 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh"] Oct 30 21:56:24.290803 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.290759 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fxv2\" (UniqueName: \"kubernetes.io/projected/015125ae-8075-4e8f-b53f-7e1a0062be96-kube-api-access-4fxv2\") pod \"maas-api-revoke-sub-809d8e0c-v2qlh\" (UID: \"015125ae-8075-4e8f-b53f-7e1a0062be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh" Oct 30 21:56:24.290995 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.290884 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsn8v\" (UniqueName: \"kubernetes.io/projected/c88fb565-e4b8-4389-852e-68b980559848-kube-api-access-fsn8v\") pod \"maas-api-revoke-sub-125b01f8-whcv6\" (UID: \"c88fb565-e4b8-4389-852e-68b980559848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6" Oct 30 21:56:24.290995 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.290983 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/015125ae-8075-4e8f-b53f-7e1a0062be96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-809d8e0c-v2qlh\" (UID: \"015125ae-8075-4e8f-b53f-7e1a0062be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh" Oct 30 21:56:24.291164 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.291144 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/015125ae-8075-4e8f-b53f-7e1a0062be96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-809d8e0c-v2qlh\" (UID: \"015125ae-8075-4e8f-b53f-7e1a0062be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh" Oct 30 21:56:24.291230 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.291213 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88fb565-e4b8-4389-852e-68b980559848-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-125b01f8-whcv6\" (UID: \"c88fb565-e4b8-4389-852e-68b980559848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6" Oct 30 21:56:24.291268 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.291260 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88fb565-e4b8-4389-852e-68b980559848-maas-api-xks-ca\") pod \"maas-api-revoke-sub-125b01f8-whcv6\" (UID: \"c88fb565-e4b8-4389-852e-68b980559848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6" Oct 30 21:56:24.392960 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.392906 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/015125ae-8075-4e8f-b53f-7e1a0062be96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-809d8e0c-v2qlh\" (UID: \"015125ae-8075-4e8f-b53f-7e1a0062be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh" Oct 30 21:56:24.393179 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.393051 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88fb565-e4b8-4389-852e-68b980559848-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-125b01f8-whcv6\" (UID: \"c88fb565-e4b8-4389-852e-68b980559848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6" Oct 30 21:56:24.393179 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.393090 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88fb565-e4b8-4389-852e-68b980559848-maas-api-xks-ca\") pod \"maas-api-revoke-sub-125b01f8-whcv6\" (UID: \"c88fb565-e4b8-4389-852e-68b980559848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6" Oct 30 21:56:24.393179 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.393138 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fxv2\" (UniqueName: \"kubernetes.io/projected/015125ae-8075-4e8f-b53f-7e1a0062be96-kube-api-access-4fxv2\") pod \"maas-api-revoke-sub-809d8e0c-v2qlh\" (UID: \"015125ae-8075-4e8f-b53f-7e1a0062be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh" Oct 30 21:56:24.393437 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.393198 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsn8v\" (UniqueName: \"kubernetes.io/projected/c88fb565-e4b8-4389-852e-68b980559848-kube-api-access-fsn8v\") pod \"maas-api-revoke-sub-125b01f8-whcv6\" (UID: \"c88fb565-e4b8-4389-852e-68b980559848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6" Oct 30 21:56:24.393437 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.393288 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/015125ae-8075-4e8f-b53f-7e1a0062be96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-809d8e0c-v2qlh\" (UID: \"015125ae-8075-4e8f-b53f-7e1a0062be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh" Oct 30 21:56:24.393799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.393770 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/015125ae-8075-4e8f-b53f-7e1a0062be96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-809d8e0c-v2qlh\" (UID: \"015125ae-8075-4e8f-b53f-7e1a0062be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh" Oct 30 21:56:24.393920 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.393893 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88fb565-e4b8-4389-852e-68b980559848-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-125b01f8-whcv6\" (UID: \"c88fb565-e4b8-4389-852e-68b980559848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6" Oct 30 21:56:24.395864 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.395842 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88fb565-e4b8-4389-852e-68b980559848-maas-api-xks-ca\") pod \"maas-api-revoke-sub-125b01f8-whcv6\" (UID: \"c88fb565-e4b8-4389-852e-68b980559848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6" Oct 30 21:56:24.396147 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.396130 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/015125ae-8075-4e8f-b53f-7e1a0062be96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-809d8e0c-v2qlh\" (UID: \"015125ae-8075-4e8f-b53f-7e1a0062be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh" Oct 30 21:56:24.410004 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.409971 4926 generic.go:356] "Generic (PLEG): container finished" podID="94e1f5fe-1324-4394-956d-8e315c4c3151" containerID="716c7df33597e2a5c22c23f352f087f5512cdc4c558154a64d0057b1f48a5389" exitCode=0 Oct 30 21:56:24.410430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.410067 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9" event={"ID":"94e1f5fe-1324-4394-956d-8e315c4c3151","Type":"ContainerDied","Data":"716c7df33597e2a5c22c23f352f087f5512cdc4c558154a64d0057b1f48a5389"} Oct 30 21:56:24.418423 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.418395 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fxv2\" (UniqueName: \"kubernetes.io/projected/015125ae-8075-4e8f-b53f-7e1a0062be96-kube-api-access-4fxv2\") pod \"maas-api-revoke-sub-809d8e0c-v2qlh\" (UID: \"015125ae-8075-4e8f-b53f-7e1a0062be96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh" Oct 30 21:56:24.418551 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.418395 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsn8v\" (UniqueName: \"kubernetes.io/projected/c88fb565-e4b8-4389-852e-68b980559848-kube-api-access-fsn8v\") pod \"maas-api-revoke-sub-125b01f8-whcv6\" (UID: \"c88fb565-e4b8-4389-852e-68b980559848\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6" Oct 30 21:56:24.511995 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.511904 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6" Oct 30 21:56:24.554124 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.554092 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh" Oct 30 21:56:24.808756 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.808609 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t"] Oct 30 21:56:24.808927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.808887 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" podUID="79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" containerName="istio-proxy" containerID="cri-o://1eb578b28777ec451fbac64f1b29a08aa0c065878298893470a9ac484bca32ae" gracePeriod=30 Oct 30 21:56:24.847826 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.847796 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6"] Oct 30 21:56:24.849433 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:56:24.849404 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88fb565_e4b8_4389_852e_68b980559848.slice/crio-96f1de17d62cc4b297fac52be1253dd934c0b82123f08d4b813a51d282bc7d4e WatchSource:0}: Error finding container 96f1de17d62cc4b297fac52be1253dd934c0b82123f08d4b813a51d282bc7d4e: Status 404 returned error can't find the container with id 96f1de17d62cc4b297fac52be1253dd934c0b82123f08d4b813a51d282bc7d4e Oct 30 21:56:24.894891 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:24.894637 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh"] Oct 30 21:56:24.896532 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:56:24.896497 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod015125ae_8075_4e8f_b53f_7e1a0062be96.slice/crio-d989286ded46a227e8f7dbcff9b0e65743e0d162866cba344a954cdbaf25ca13 WatchSource:0}: Error finding container d989286ded46a227e8f7dbcff9b0e65743e0d162866cba344a954cdbaf25ca13: Status 404 returned error can't find the container with id d989286ded46a227e8f7dbcff9b0e65743e0d162866cba344a954cdbaf25ca13 Oct 30 21:56:25.259402 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.259353 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" containerID="cri-o://7a04844314a11e178a8ab24fe805478b79cf01f25e0cd2734da35498e2e70a75" gracePeriod=45 Oct 30 21:56:25.268623 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.268579 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" containerID="cri-o://9ea6e4b39b51bee1c5f01f2ad7c1d130ecf5239b88bcb5ddb6b81d970e21a267" gracePeriod=45 Oct 30 21:56:25.273924 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.273877 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" containerID="cri-o://a8e8991031872ef634a232d625e7470c8539b3ad1f885726de699b922926aaf9" gracePeriod=45 Oct 30 21:56:25.369408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.369376 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq"] Oct 30 21:56:25.374255 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.374120 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq" Oct 30 21:56:25.394089 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.393838 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq"] Oct 30 21:56:25.432328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.432159 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3117a72-cb57-4508-ac1d-e69000eb7da5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq\" (UID: \"d3117a72-cb57-4508-ac1d-e69000eb7da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq" Oct 30 21:56:25.432328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.432282 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3117a72-cb57-4508-ac1d-e69000eb7da5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq\" (UID: \"d3117a72-cb57-4508-ac1d-e69000eb7da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq" Oct 30 21:56:25.432877 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.432437 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md7tn\" (UniqueName: \"kubernetes.io/projected/d3117a72-cb57-4508-ac1d-e69000eb7da5-kube-api-access-md7tn\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq\" (UID: \"d3117a72-cb57-4508-ac1d-e69000eb7da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq" Oct 30 21:56:25.435084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.434378 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6" event={"ID":"c88fb565-e4b8-4389-852e-68b980559848","Type":"ContainerStarted","Data":"a1e4b9e381dd938f5f131b186384c9711484929e13da541ba4d7a6a274990120"} Oct 30 21:56:25.435084 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.434481 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6" event={"ID":"c88fb565-e4b8-4389-852e-68b980559848","Type":"ContainerStarted","Data":"96f1de17d62cc4b297fac52be1253dd934c0b82123f08d4b813a51d282bc7d4e"} Oct 30 21:56:25.438298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.438223 4926 generic.go:356] "Generic (PLEG): container finished" podID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerID="7a04844314a11e178a8ab24fe805478b79cf01f25e0cd2734da35498e2e70a75" exitCode=0 Oct 30 21:56:25.438298 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.438286 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" event={"ID":"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f","Type":"ContainerDied","Data":"7a04844314a11e178a8ab24fe805478b79cf01f25e0cd2734da35498e2e70a75"} Oct 30 21:56:25.440755 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.440734 4926 generic.go:356] "Generic (PLEG): container finished" podID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerID="a8e8991031872ef634a232d625e7470c8539b3ad1f885726de699b922926aaf9" exitCode=0 Oct 30 21:56:25.441069 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.440904 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" event={"ID":"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32","Type":"ContainerDied","Data":"a8e8991031872ef634a232d625e7470c8539b3ad1f885726de699b922926aaf9"} Oct 30 21:56:25.446025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.445916 4926 generic.go:356] "Generic (PLEG): container finished" podID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerID="9ea6e4b39b51bee1c5f01f2ad7c1d130ecf5239b88bcb5ddb6b81d970e21a267" exitCode=0 Oct 30 21:56:25.446025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.445991 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" event={"ID":"81b30160-e47f-45ea-8d4c-c6450c1e86bd","Type":"ContainerDied","Data":"9ea6e4b39b51bee1c5f01f2ad7c1d130ecf5239b88bcb5ddb6b81d970e21a267"} Oct 30 21:56:25.448783 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.448707 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh" event={"ID":"015125ae-8075-4e8f-b53f-7e1a0062be96","Type":"ContainerStarted","Data":"39c01caad133359a2535fff23a6bf1665a37bd50e6fc93c5205e449d7c4c9c5e"} Oct 30 21:56:25.448783 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.448747 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh" event={"ID":"015125ae-8075-4e8f-b53f-7e1a0062be96","Type":"ContainerStarted","Data":"d989286ded46a227e8f7dbcff9b0e65743e0d162866cba344a954cdbaf25ca13"} Oct 30 21:56:25.459126 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.459094 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.87:8000/ready\": dial tcp 10.132.0.87:8000: connect: connection refused" Oct 30 21:56:25.547253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.535980 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3117a72-cb57-4508-ac1d-e69000eb7da5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq\" (UID: \"d3117a72-cb57-4508-ac1d-e69000eb7da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq" Oct 30 21:56:25.547253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.540319 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3117a72-cb57-4508-ac1d-e69000eb7da5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq\" (UID: \"d3117a72-cb57-4508-ac1d-e69000eb7da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq" Oct 30 21:56:25.547253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.541612 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3117a72-cb57-4508-ac1d-e69000eb7da5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq\" (UID: \"d3117a72-cb57-4508-ac1d-e69000eb7da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq" Oct 30 21:56:25.547253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.541769 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md7tn\" (UniqueName: \"kubernetes.io/projected/d3117a72-cb57-4508-ac1d-e69000eb7da5-kube-api-access-md7tn\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq\" (UID: \"d3117a72-cb57-4508-ac1d-e69000eb7da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq" Oct 30 21:56:25.547253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.542530 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6" podStartSLOduration=1.5425134539999998 podStartE2EDuration="1.542513454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:25.494565976 +0000 UTC m=+3372.615639722" watchObservedRunningTime="2026-10-30 21:56:25.542513454 +0000 UTC m=+3372.663587201" Oct 30 21:56:25.547253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.543130 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh" podStartSLOduration=1.5431184199999999 podStartE2EDuration="1.54311842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:25.540895707 +0000 UTC m=+3372.661969454" watchObservedRunningTime="2026-10-30 21:56:25.54311842 +0000 UTC m=+3372.664192165" Oct 30 21:56:25.547253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.543818 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3117a72-cb57-4508-ac1d-e69000eb7da5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq\" (UID: \"d3117a72-cb57-4508-ac1d-e69000eb7da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq" Oct 30 21:56:25.565576 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.565199 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md7tn\" (UniqueName: \"kubernetes.io/projected/d3117a72-cb57-4508-ac1d-e69000eb7da5-kube-api-access-md7tn\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq\" (UID: \"d3117a72-cb57-4508-ac1d-e69000eb7da5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq" Oct 30 21:56:25.608096 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.608068 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:56:25.692220 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.686965 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:56:25.744301 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.744269 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-dshm\") pod \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " Oct 30 21:56:25.744469 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.744326 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-dshm\") pod \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " Oct 30 21:56:25.744469 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.744408 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81b30160-e47f-45ea-8d4c-c6450c1e86bd-tls-certs\" (UniqueName: \"kubernetes.io/secret/81b30160-e47f-45ea-8d4c-c6450c1e86bd-tls-certs\") pod \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " Oct 30 21:56:25.744469 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.744439 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-home\" (UniqueName: \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-home\") pod \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " Oct 30 21:56:25.744641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.744492 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b30160-e47f-45ea-8d4c-c6450c1e86bd-kube-api-access-p4k54\" (UniqueName: \"kubernetes.io/projected/81b30160-e47f-45ea-8d4c-c6450c1e86bd-kube-api-access-p4k54\") pod \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " Oct 30 21:56:25.744641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.744527 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-tmp-dir\") pod \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " Oct 30 21:56:25.744641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.744578 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-model-cache\") pod \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " Oct 30 21:56:25.744641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.744613 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-tmp-dir\") pod \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\" (UID: \"81b30160-e47f-45ea-8d4c-c6450c1e86bd\") " Oct 30 21:56:25.744876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.744640 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-model-cache\") pod \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " Oct 30 21:56:25.744876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.744704 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-kube-api-access-sbhfh\" (UniqueName: \"kubernetes.io/projected/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-kube-api-access-sbhfh\") pod \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " Oct 30 21:56:25.744876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.744815 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-home\" (UniqueName: \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-home\") pod \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " Oct 30 21:56:25.745049 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.744889 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-tls-certs\") pod \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\" (UID: \"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32\") " Oct 30 21:56:25.745894 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.745817 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-home" pod "81b30160-e47f-45ea-8d4c-c6450c1e86bd" (UID: "81b30160-e47f-45ea-8d4c-c6450c1e86bd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:25.746268 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.746074 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-tmp-dir" pod "81b30160-e47f-45ea-8d4c-c6450c1e86bd" (UID: "81b30160-e47f-45ea-8d4c-c6450c1e86bd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:25.746379 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.746313 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-model-cache" pod "f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" (UID: "f1f7f668-8bb1-435b-a2d8-7340fc3b7c32"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:25.748775 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.746562 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-tmp-dir" pod "f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" (UID: "f1f7f668-8bb1-435b-a2d8-7340fc3b7c32"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:25.748775 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.748386 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-dshm" pod "81b30160-e47f-45ea-8d4c-c6450c1e86bd" (UID: "81b30160-e47f-45ea-8d4c-c6450c1e86bd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:25.749004 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.748972 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-dshm" pod "f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" (UID: "f1f7f668-8bb1-435b-a2d8-7340fc3b7c32"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:25.749390 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.749365 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-model-cache" pod "81b30160-e47f-45ea-8d4c-c6450c1e86bd" (UID: "81b30160-e47f-45ea-8d4c-c6450c1e86bd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:25.750026 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.749989 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-kube-api-access-sbhfh" pod "f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" (UID: "f1f7f668-8bb1-435b-a2d8-7340fc3b7c32"). InnerVolumeSpecName "kube-api-access-sbhfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:25.751150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.751124 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-home" pod "f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" (UID: "f1f7f668-8bb1-435b-a2d8-7340fc3b7c32"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:25.752228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.752194 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b30160-e47f-45ea-8d4c-c6450c1e86bd-tls-certs" pod "81b30160-e47f-45ea-8d4c-c6450c1e86bd" (UID: "81b30160-e47f-45ea-8d4c-c6450c1e86bd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:25.752449 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.752422 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b30160-e47f-45ea-8d4c-c6450c1e86bd-kube-api-access-p4k54" pod "81b30160-e47f-45ea-8d4c-c6450c1e86bd" (UID: "81b30160-e47f-45ea-8d4c-c6450c1e86bd"). InnerVolumeSpecName "kube-api-access-p4k54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:25.752576 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.752555 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-tls-certs" pod "f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" (UID: "f1f7f668-8bb1-435b-a2d8-7340fc3b7c32"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:25.757199 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.757176 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:56:25.768495 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.768257 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq" Oct 30 21:56:25.847126 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.847076 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-kube-api-access-nk692\" (UniqueName: \"kubernetes.io/projected/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-kube-api-access-nk692\") pod \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " Oct 30 21:56:25.847326 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.847179 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-tls-certs\" (UniqueName: \"kubernetes.io/secret/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-tls-certs\") pod \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " Oct 30 21:56:25.847326 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.847219 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-home\" (UniqueName: \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-home\") pod \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " Oct 30 21:56:25.847472 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.847388 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-model-cache\") pod \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " Oct 30 21:56:25.847533 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.847471 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-home" pod "fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" (UID: "fc449fbe-a012-44a2-b57b-5ab7bbe1f44f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:25.847533 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.847489 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-tmp-dir\") pod \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " Oct 30 21:56:25.847667 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.847649 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-dshm\") pod \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\" (UID: \"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f\") " Oct 30 21:56:25.847768 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.847726 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-model-cache" pod "fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" (UID: "fc449fbe-a012-44a2-b57b-5ab7bbe1f44f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:25.847900 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.847878 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-tmp-dir" pod "fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" (UID: "fc449fbe-a012-44a2-b57b-5ab7bbe1f44f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:25.848340 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.848323 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:25.848459 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.848449 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b30160-e47f-45ea-8d4c-c6450c1e86bd-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:25.848520 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.848513 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:25.848610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.848599 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4k54\" (UniqueName: \"kubernetes.io/projected/81b30160-e47f-45ea-8d4c-c6450c1e86bd-kube-api-access-p4k54\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:25.848690 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.848677 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:25.849016 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.849004 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:25.849322 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.849304 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:25.849438 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.849425 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:25.849545 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.849533 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbhfh\" (UniqueName: \"kubernetes.io/projected/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-kube-api-access-sbhfh\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:25.849637 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.849625 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:25.849742 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.849729 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:25.849839 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.849825 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:25.850246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.850087 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:25.850516 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.850502 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:25.850641 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.850628 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b30160-e47f-45ea-8d4c-c6450c1e86bd-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:25.851768 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.851683 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-dshm" pod "fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" (UID: "fc449fbe-a012-44a2-b57b-5ab7bbe1f44f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:25.851858 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.851748 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-tls-certs" pod "fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" (UID: "fc449fbe-a012-44a2-b57b-5ab7bbe1f44f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:25.854330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.854307 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-kube-api-access-nk692" pod "fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" (UID: "fc449fbe-a012-44a2-b57b-5ab7bbe1f44f"). InnerVolumeSpecName "kube-api-access-nk692". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:25.854832 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.854817 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9" Oct 30 21:56:25.951493 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.951459 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e1f5fe-1324-4394-956d-8e315c4c3151-kube-api-access-9wblj\" (UniqueName: \"kubernetes.io/projected/94e1f5fe-1324-4394-956d-8e315c4c3151-kube-api-access-9wblj\") pod \"94e1f5fe-1324-4394-956d-8e315c4c3151\" (UID: \"94e1f5fe-1324-4394-956d-8e315c4c3151\") " Oct 30 21:56:25.951666 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.951517 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94e1f5fe-1324-4394-956d-8e315c4c3151-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e1f5fe-1324-4394-956d-8e315c4c3151-maas-api-xks-ca\") pod \"94e1f5fe-1324-4394-956d-8e315c4c3151\" (UID: \"94e1f5fe-1324-4394-956d-8e315c4c3151\") " Oct 30 21:56:25.951666 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.951587 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94e1f5fe-1324-4394-956d-8e315c4c3151-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e1f5fe-1324-4394-956d-8e315c4c3151-maas-api-openshift-ca\") pod \"94e1f5fe-1324-4394-956d-8e315c4c3151\" (UID: \"94e1f5fe-1324-4394-956d-8e315c4c3151\") " Oct 30 21:56:25.952044 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.951995 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e1f5fe-1324-4394-956d-8e315c4c3151-maas-api-openshift-ca" pod "94e1f5fe-1324-4394-956d-8e315c4c3151" (UID: "94e1f5fe-1324-4394-956d-8e315c4c3151"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:25.952245 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.952224 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:25.952315 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.952251 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94e1f5fe-1324-4394-956d-8e315c4c3151-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:25.952315 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.952268 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk692\" (UniqueName: \"kubernetes.io/projected/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-kube-api-access-nk692\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:25.952315 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.952283 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:25.953660 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.953630 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94e1f5fe-1324-4394-956d-8e315c4c3151-maas-api-xks-ca" pod "94e1f5fe-1324-4394-956d-8e315c4c3151" (UID: "94e1f5fe-1324-4394-956d-8e315c4c3151"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:25.953827 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:25.953807 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e1f5fe-1324-4394-956d-8e315c4c3151-kube-api-access-9wblj" pod "94e1f5fe-1324-4394-956d-8e315c4c3151" (UID: "94e1f5fe-1324-4394-956d-8e315c4c3151"). InnerVolumeSpecName "kube-api-access-9wblj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:26.053466 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.053432 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wblj\" (UniqueName: \"kubernetes.io/projected/94e1f5fe-1324-4394-956d-8e315c4c3151-kube-api-access-9wblj\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:26.053466 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.053465 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94e1f5fe-1324-4394-956d-8e315c4c3151-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:26.161677 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.161636 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq"] Oct 30 21:56:26.161853 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:56:26.161826 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3117a72_cb57_4508_ac1d_e69000eb7da5.slice/crio-e11502230f312bedaa2f5ada7e769066fc0ce800d93908e8895c1f20a9530213 WatchSource:0}: Error finding container e11502230f312bedaa2f5ada7e769066fc0ce800d93908e8895c1f20a9530213: Status 404 returned error can't find the container with id e11502230f312bedaa2f5ada7e769066fc0ce800d93908e8895c1f20a9530213 Oct 30 21:56:26.457424 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.457383 4926 generic.go:356] "Generic (PLEG): container finished" podID="c88fb565-e4b8-4389-852e-68b980559848" containerID="a1e4b9e381dd938f5f131b186384c9711484929e13da541ba4d7a6a274990120" exitCode=0 Oct 30 21:56:26.457891 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.457463 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6" event={"ID":"c88fb565-e4b8-4389-852e-68b980559848","Type":"ContainerDied","Data":"a1e4b9e381dd938f5f131b186384c9711484929e13da541ba4d7a6a274990120"} Oct 30 21:56:26.458906 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.458877 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq" event={"ID":"d3117a72-cb57-4508-ac1d-e69000eb7da5","Type":"ContainerStarted","Data":"e11502230f312bedaa2f5ada7e769066fc0ce800d93908e8895c1f20a9530213"} Oct 30 21:56:26.460284 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.460264 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" Oct 30 21:56:26.460386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.460292 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2" event={"ID":"fc449fbe-a012-44a2-b57b-5ab7bbe1f44f","Type":"ContainerDied","Data":"d17d47553fccaceea09b6d2c843a862949237437e241e89537a7bec21811e156"} Oct 30 21:56:26.460386 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.460342 4926 scope.go:122] "RemoveContainer" containerID="7a04844314a11e178a8ab24fe805478b79cf01f25e0cd2734da35498e2e70a75" Oct 30 21:56:26.461816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.461798 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9" Oct 30 21:56:26.461928 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.461802 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9" event={"ID":"94e1f5fe-1324-4394-956d-8e315c4c3151","Type":"ContainerDied","Data":"d37d3bae05e002b94441d3f1b87c1b871a600670e10de739b04ba4db0c9112a0"} Oct 30 21:56:26.462010 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.461943 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d37d3bae05e002b94441d3f1b87c1b871a600670e10de739b04ba4db0c9112a0" Oct 30 21:56:26.463466 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.463449 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" Oct 30 21:56:26.463566 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.463489 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" event={"ID":"f1f7f668-8bb1-435b-a2d8-7340fc3b7c32","Type":"ContainerDied","Data":"e25e0d45311a0842c82b962ffd57a5e9efde7616cdaa125200239d44f13e16c7"} Oct 30 21:56:26.465347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.465329 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" Oct 30 21:56:26.465449 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.465352 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg" event={"ID":"81b30160-e47f-45ea-8d4c-c6450c1e86bd","Type":"ContainerDied","Data":"fa1ab83fc155845a5bdfecdb923f61888ea9a3968db0361fb1bafd2028a4be4b"} Oct 30 21:56:26.467020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.466998 4926 generic.go:356] "Generic (PLEG): container finished" podID="015125ae-8075-4e8f-b53f-7e1a0062be96" containerID="39c01caad133359a2535fff23a6bf1665a37bd50e6fc93c5205e449d7c4c9c5e" exitCode=0 Oct 30 21:56:26.467225 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.467063 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh" event={"ID":"015125ae-8075-4e8f-b53f-7e1a0062be96","Type":"ContainerDied","Data":"39c01caad133359a2535fff23a6bf1665a37bd50e6fc93c5205e449d7c4c9c5e"} Oct 30 21:56:26.479098 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.479078 4926 scope.go:122] "RemoveContainer" containerID="a8e8991031872ef634a232d625e7470c8539b3ad1f885726de699b922926aaf9" Oct 30 21:56:26.493614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.493589 4926 scope.go:122] "RemoveContainer" containerID="9ea6e4b39b51bee1c5f01f2ad7c1d130ecf5239b88bcb5ddb6b81d970e21a267" Oct 30 21:56:26.550844 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.550805 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg"] Oct 30 21:56:26.557083 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.557042 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-56ac74/premium-w1-56ac74-kserve-54d4647957-4lcpg"] Oct 30 21:56:26.558279 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.558242 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 30 21:56:26.671778 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.671730 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2"] Oct 30 21:56:26.687001 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.686907 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-56ac74/distinct-2-w1-56ac74-kserve-8b899879f-9nrt2"] Oct 30 21:56:26.726837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.726796 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws"] Oct 30 21:56:26.731687 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:26.731650 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-56ac74/simulator-w1-56ac74-kserve-df666c46d-pw8ws"] Oct 30 21:56:27.016823 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.016736 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" path="/var/lib/kubelet/pods/81b30160-e47f-45ea-8d4c-c6450c1e86bd/volumes" Oct 30 21:56:27.017420 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.017401 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" path="/var/lib/kubelet/pods/f1f7f668-8bb1-435b-a2d8-7340fc3b7c32/volumes" Oct 30 21:56:27.017988 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.017971 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" path="/var/lib/kubelet/pods/fc449fbe-a012-44a2-b57b-5ab7bbe1f44f/volumes" Oct 30 21:56:27.477151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.477116 4926 generic.go:356] "Generic (PLEG): container finished" podID="d3117a72-cb57-4508-ac1d-e69000eb7da5" containerID="ead6b8974fd43ad35b4c9691fc0fb442a41540bb43c8192c13c756f95025b147" exitCode=0 Oct 30 21:56:27.477683 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.477213 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq" event={"ID":"d3117a72-cb57-4508-ac1d-e69000eb7da5","Type":"ContainerDied","Data":"ead6b8974fd43ad35b4c9691fc0fb442a41540bb43c8192c13c756f95025b147"} Oct 30 21:56:27.664155 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.664121 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6" Oct 30 21:56:27.695228 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.695198 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh" Oct 30 21:56:27.780568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.780464 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c88fb565-e4b8-4389-852e-68b980559848-kube-api-access-fsn8v\" (UniqueName: \"kubernetes.io/projected/c88fb565-e4b8-4389-852e-68b980559848-kube-api-access-fsn8v\") pod \"c88fb565-e4b8-4389-852e-68b980559848\" (UID: \"c88fb565-e4b8-4389-852e-68b980559848\") " Oct 30 21:56:27.780771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.780593 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/015125ae-8075-4e8f-b53f-7e1a0062be96-kube-api-access-4fxv2\" (UniqueName: \"kubernetes.io/projected/015125ae-8075-4e8f-b53f-7e1a0062be96-kube-api-access-4fxv2\") pod \"015125ae-8075-4e8f-b53f-7e1a0062be96\" (UID: \"015125ae-8075-4e8f-b53f-7e1a0062be96\") " Oct 30 21:56:27.780771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.780661 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/015125ae-8075-4e8f-b53f-7e1a0062be96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/015125ae-8075-4e8f-b53f-7e1a0062be96-maas-api-openshift-ca\") pod \"015125ae-8075-4e8f-b53f-7e1a0062be96\" (UID: \"015125ae-8075-4e8f-b53f-7e1a0062be96\") " Oct 30 21:56:27.780771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.780688 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c88fb565-e4b8-4389-852e-68b980559848-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88fb565-e4b8-4389-852e-68b980559848-maas-api-xks-ca\") pod \"c88fb565-e4b8-4389-852e-68b980559848\" (UID: \"c88fb565-e4b8-4389-852e-68b980559848\") " Oct 30 21:56:27.780771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.780758 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/015125ae-8075-4e8f-b53f-7e1a0062be96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/015125ae-8075-4e8f-b53f-7e1a0062be96-maas-api-xks-ca\") pod \"015125ae-8075-4e8f-b53f-7e1a0062be96\" (UID: \"015125ae-8075-4e8f-b53f-7e1a0062be96\") " Oct 30 21:56:27.781005 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.780832 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c88fb565-e4b8-4389-852e-68b980559848-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88fb565-e4b8-4389-852e-68b980559848-maas-api-openshift-ca\") pod \"c88fb565-e4b8-4389-852e-68b980559848\" (UID: \"c88fb565-e4b8-4389-852e-68b980559848\") " Oct 30 21:56:27.781636 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.781605 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c88fb565-e4b8-4389-852e-68b980559848-maas-api-openshift-ca" pod "c88fb565-e4b8-4389-852e-68b980559848" (UID: "c88fb565-e4b8-4389-852e-68b980559848"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:27.781859 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.781828 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/015125ae-8075-4e8f-b53f-7e1a0062be96-maas-api-openshift-ca" pod "015125ae-8075-4e8f-b53f-7e1a0062be96" (UID: "015125ae-8075-4e8f-b53f-7e1a0062be96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:27.783355 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.783327 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88fb565-e4b8-4389-852e-68b980559848-kube-api-access-fsn8v" pod "c88fb565-e4b8-4389-852e-68b980559848" (UID: "c88fb565-e4b8-4389-852e-68b980559848"). InnerVolumeSpecName "kube-api-access-fsn8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:27.784212 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.784177 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/015125ae-8075-4e8f-b53f-7e1a0062be96-maas-api-xks-ca" pod "015125ae-8075-4e8f-b53f-7e1a0062be96" (UID: "015125ae-8075-4e8f-b53f-7e1a0062be96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:27.784311 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.784293 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c88fb565-e4b8-4389-852e-68b980559848-maas-api-xks-ca" pod "c88fb565-e4b8-4389-852e-68b980559848" (UID: "c88fb565-e4b8-4389-852e-68b980559848"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:27.785204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.785176 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/015125ae-8075-4e8f-b53f-7e1a0062be96-kube-api-access-4fxv2" pod "015125ae-8075-4e8f-b53f-7e1a0062be96" (UID: "015125ae-8075-4e8f-b53f-7e1a0062be96"). InnerVolumeSpecName "kube-api-access-4fxv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:27.882673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.882626 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fxv2\" (UniqueName: \"kubernetes.io/projected/015125ae-8075-4e8f-b53f-7e1a0062be96-kube-api-access-4fxv2\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:27.882673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.882670 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/015125ae-8075-4e8f-b53f-7e1a0062be96-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:27.882948 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.882685 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c88fb565-e4b8-4389-852e-68b980559848-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:27.882948 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.882700 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/015125ae-8075-4e8f-b53f-7e1a0062be96-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:27.882948 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.882714 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c88fb565-e4b8-4389-852e-68b980559848-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:27.882948 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:27.882727 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsn8v\" (UniqueName: \"kubernetes.io/projected/c88fb565-e4b8-4389-852e-68b980559848-kube-api-access-fsn8v\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:28.491805 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:28.491769 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6" Oct 30 21:56:28.492344 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:28.491796 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6" event={"ID":"c88fb565-e4b8-4389-852e-68b980559848","Type":"ContainerDied","Data":"96f1de17d62cc4b297fac52be1253dd934c0b82123f08d4b813a51d282bc7d4e"} Oct 30 21:56:28.492344 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:28.491857 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96f1de17d62cc4b297fac52be1253dd934c0b82123f08d4b813a51d282bc7d4e" Oct 30 21:56:28.493782 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:28.493756 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh" Oct 30 21:56:28.493782 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:28.493771 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh" event={"ID":"015125ae-8075-4e8f-b53f-7e1a0062be96","Type":"ContainerDied","Data":"d989286ded46a227e8f7dbcff9b0e65743e0d162866cba344a954cdbaf25ca13"} Oct 30 21:56:28.493992 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:28.493812 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d989286ded46a227e8f7dbcff9b0e65743e0d162866cba344a954cdbaf25ca13" Oct 30 21:56:28.649665 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:28.649635 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq" Oct 30 21:56:28.794330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:28.794223 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3117a72-cb57-4508-ac1d-e69000eb7da5-kube-api-access-md7tn\" (UniqueName: \"kubernetes.io/projected/d3117a72-cb57-4508-ac1d-e69000eb7da5-kube-api-access-md7tn\") pod \"d3117a72-cb57-4508-ac1d-e69000eb7da5\" (UID: \"d3117a72-cb57-4508-ac1d-e69000eb7da5\") " Oct 30 21:56:28.794330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:28.794302 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3117a72-cb57-4508-ac1d-e69000eb7da5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3117a72-cb57-4508-ac1d-e69000eb7da5-maas-api-openshift-ca\") pod \"d3117a72-cb57-4508-ac1d-e69000eb7da5\" (UID: \"d3117a72-cb57-4508-ac1d-e69000eb7da5\") " Oct 30 21:56:28.794565 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:28.794428 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3117a72-cb57-4508-ac1d-e69000eb7da5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3117a72-cb57-4508-ac1d-e69000eb7da5-maas-api-xks-ca\") pod \"d3117a72-cb57-4508-ac1d-e69000eb7da5\" (UID: \"d3117a72-cb57-4508-ac1d-e69000eb7da5\") " Oct 30 21:56:28.794848 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:28.794817 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3117a72-cb57-4508-ac1d-e69000eb7da5-maas-api-openshift-ca" pod "d3117a72-cb57-4508-ac1d-e69000eb7da5" (UID: "d3117a72-cb57-4508-ac1d-e69000eb7da5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:28.797073 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:28.797027 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3117a72-cb57-4508-ac1d-e69000eb7da5-maas-api-xks-ca" pod "d3117a72-cb57-4508-ac1d-e69000eb7da5" (UID: "d3117a72-cb57-4508-ac1d-e69000eb7da5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:28.797192 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:28.797091 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3117a72-cb57-4508-ac1d-e69000eb7da5-kube-api-access-md7tn" pod "d3117a72-cb57-4508-ac1d-e69000eb7da5" (UID: "d3117a72-cb57-4508-ac1d-e69000eb7da5"). InnerVolumeSpecName "kube-api-access-md7tn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:28.896366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:28.896313 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3117a72-cb57-4508-ac1d-e69000eb7da5-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:28.896366 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:28.896365 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md7tn\" (UniqueName: \"kubernetes.io/projected/d3117a72-cb57-4508-ac1d-e69000eb7da5-kube-api-access-md7tn\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:28.896606 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:28.896384 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3117a72-cb57-4508-ac1d-e69000eb7da5-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:29.057446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:29.057358 4926 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod57902142-df3e-4114-ac6c-7d42a300f2f9"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod57902142-df3e-4114-ac6c-7d42a300f2f9] : Timed out while waiting for systemd to remove kubepods-burstable-pod57902142_df3e_4114_ac6c_7d42a300f2f9.slice" Oct 30 21:56:29.057446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:29.057398 4926 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="57902142-df3e-4114-ac6c-7d42a300f2f9" err="failed to delete cgroup paths for [kubepods burstable pod57902142-df3e-4114-ac6c-7d42a300f2f9] : unable to destroy cgroup paths for cgroup [kubepods burstable pod57902142-df3e-4114-ac6c-7d42a300f2f9] : Timed out while waiting for systemd to remove kubepods-burstable-pod57902142_df3e_4114_ac6c_7d42a300f2f9.slice" Oct 30 21:56:29.505427 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:29.505385 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq" Oct 30 21:56:29.505831 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:29.505422 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq" event={"ID":"d3117a72-cb57-4508-ac1d-e69000eb7da5","Type":"ContainerDied","Data":"e11502230f312bedaa2f5ada7e769066fc0ce800d93908e8895c1f20a9530213"} Oct 30 21:56:29.505831 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:29.505520 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e11502230f312bedaa2f5ada7e769066fc0ce800d93908e8895c1f20a9530213" Oct 30 21:56:30.091115 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.090677 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:56:30.216094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.216053 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-podinfo\") pod \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " Oct 30 21:56:30.216308 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.216106 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istiod-ca-cert\") pod \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " Oct 30 21:56:30.216308 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.216155 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-credential-socket\") pod \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " Oct 30 21:56:30.216308 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.216194 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-envoy\") pod \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " Oct 30 21:56:30.216308 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.216289 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-workload-socket\") pod \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " Oct 30 21:56:30.216810 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.216335 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-token\" (UniqueName: \"kubernetes.io/projected/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-token\") pod \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " Oct 30 21:56:30.216810 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.216376 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-workload-certs\") pod \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " Oct 30 21:56:30.216810 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.216435 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-kube-api-access-rtw7d\" (UniqueName: \"kubernetes.io/projected/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-kube-api-access-rtw7d\") pod \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " Oct 30 21:56:30.216810 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.216495 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-data\") pod \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\" (UID: \"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee\") " Oct 30 21:56:30.216810 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.216580 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-credential-socket" pod "79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" (UID: "79ee4f20-47b0-4c7a-9ad1-ec970173c3ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:30.217114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.216818 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-workload-certs" pod "79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" (UID: "79ee4f20-47b0-4c7a-9ad1-ec970173c3ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:30.217227 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.217011 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-workload-socket" pod "79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" (UID: "79ee4f20-47b0-4c7a-9ad1-ec970173c3ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:30.217288 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.217024 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-data" pod "79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" (UID: "79ee4f20-47b0-4c7a-9ad1-ec970173c3ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:30.217288 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.217120 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:30.217288 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.217278 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:30.217436 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.217336 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istiod-ca-cert" pod "79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" (UID: "79ee4f20-47b0-4c7a-9ad1-ec970173c3ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:30.218614 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.218587 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-podinfo" pod "79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" (UID: "79ee4f20-47b0-4c7a-9ad1-ec970173c3ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:56:30.219156 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.219133 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-kube-api-access-rtw7d" pod "79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" (UID: "79ee4f20-47b0-4c7a-9ad1-ec970173c3ee"). InnerVolumeSpecName "kube-api-access-rtw7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:30.219890 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.219869 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-envoy" pod "79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" (UID: "79ee4f20-47b0-4c7a-9ad1-ec970173c3ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:56:30.219991 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.219906 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-token" pod "79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" (UID: "79ee4f20-47b0-4c7a-9ad1-ec970173c3ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:30.318331 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.318295 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:30.318331 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.318328 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:30.318331 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.318337 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:30.318585 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.318345 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:30.318585 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.318358 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtw7d\" (UniqueName: \"kubernetes.io/projected/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-kube-api-access-rtw7d\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:30.318585 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.318371 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:30.318585 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.318379 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:30.516511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.516467 4926 generic.go:356] "Generic (PLEG): container finished" podID="79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" containerID="1eb578b28777ec451fbac64f1b29a08aa0c065878298893470a9ac484bca32ae" exitCode=0 Oct 30 21:56:30.516511 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.516514 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" Oct 30 21:56:30.517072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.516548 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" event={"ID":"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee","Type":"ContainerDied","Data":"1eb578b28777ec451fbac64f1b29a08aa0c065878298893470a9ac484bca32ae"} Oct 30 21:56:30.517072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.516607 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t" event={"ID":"79ee4f20-47b0-4c7a-9ad1-ec970173c3ee","Type":"ContainerDied","Data":"d431cbbbcf01501a3567817c414081b6f34db1f43ce7cc9db79576a6ecaa712a"} Oct 30 21:56:30.517072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.516638 4926 scope.go:122] "RemoveContainer" containerID="1eb578b28777ec451fbac64f1b29a08aa0c065878298893470a9ac484bca32ae" Oct 30 21:56:30.528403 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.528378 4926 scope.go:122] "RemoveContainer" containerID="db7bc3b531fa7b313086aaf969b710e52a1e7b3b6ece5c8f378e2767ccef200e" Oct 30 21:56:30.540245 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.540224 4926 scope.go:122] "RemoveContainer" containerID="1eb578b28777ec451fbac64f1b29a08aa0c065878298893470a9ac484bca32ae" Oct 30 21:56:30.540556 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:56:30.540538 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1eb578b28777ec451fbac64f1b29a08aa0c065878298893470a9ac484bca32ae\": container with ID starting with 1eb578b28777ec451fbac64f1b29a08aa0c065878298893470a9ac484bca32ae not found: ID does not exist" containerID="1eb578b28777ec451fbac64f1b29a08aa0c065878298893470a9ac484bca32ae" Oct 30 21:56:30.540634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.540570 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1eb578b28777ec451fbac64f1b29a08aa0c065878298893470a9ac484bca32ae"} err="failed to get container status \"1eb578b28777ec451fbac64f1b29a08aa0c065878298893470a9ac484bca32ae\": rpc error: code = NotFound desc = could not find container \"1eb578b28777ec451fbac64f1b29a08aa0c065878298893470a9ac484bca32ae\": container with ID starting with 1eb578b28777ec451fbac64f1b29a08aa0c065878298893470a9ac484bca32ae not found: ID does not exist" Oct 30 21:56:30.540634 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.540596 4926 scope.go:122] "RemoveContainer" containerID="db7bc3b531fa7b313086aaf969b710e52a1e7b3b6ece5c8f378e2767ccef200e" Oct 30 21:56:30.540867 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:56:30.540847 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db7bc3b531fa7b313086aaf969b710e52a1e7b3b6ece5c8f378e2767ccef200e\": container with ID starting with db7bc3b531fa7b313086aaf969b710e52a1e7b3b6ece5c8f378e2767ccef200e not found: ID does not exist" containerID="db7bc3b531fa7b313086aaf969b710e52a1e7b3b6ece5c8f378e2767ccef200e" Oct 30 21:56:30.540928 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.540873 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7bc3b531fa7b313086aaf969b710e52a1e7b3b6ece5c8f378e2767ccef200e"} err="failed to get container status \"db7bc3b531fa7b313086aaf969b710e52a1e7b3b6ece5c8f378e2767ccef200e\": rpc error: code = NotFound desc = could not find container \"db7bc3b531fa7b313086aaf969b710e52a1e7b3b6ece5c8f378e2767ccef200e\": container with ID starting with db7bc3b531fa7b313086aaf969b710e52a1e7b3b6ece5c8f378e2767ccef200e not found: ID does not exist" Oct 30 21:56:30.579420 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.579382 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t"] Oct 30 21:56:30.582837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:30.582804 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f8137c-openshift-default-7fdc8fc477-nlf5t"] Oct 30 21:56:31.018626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:31.018592 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" path="/var/lib/kubelet/pods/79ee4f20-47b0-4c7a-9ad1-ec970173c3ee/volumes" Oct 30 21:56:31.060253 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:31.060214 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv"] Oct 30 21:56:31.072937 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:31.072899 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f80b7bf0-zrfhv"] Oct 30 21:56:31.308678 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:31.308582 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:56:33.015663 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.015627 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6768e51-6568-4cd7-a64d-0ae77e339ce0" path="/var/lib/kubelet/pods/a6768e51-6568-4cd7-a64d-0ae77e339ce0/volumes" Oct 30 21:56:33.817918 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.817879 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9"] Oct 30 21:56:33.818411 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818397 4926 cpu_manager.go:414] "containerMap: removing container" podUID="79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" containerName="istio-proxy" Oct 30 21:56:33.818458 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818413 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" containerName="istio-proxy" Oct 30 21:56:33.818458 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818427 4926 cpu_manager.go:414] "containerMap: removing container" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" Oct 30 21:56:33.818458 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818432 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" Oct 30 21:56:33.818458 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818454 4926 cpu_manager.go:414] "containerMap: removing container" podUID="015125ae-8075-4e8f-b53f-7e1a0062be96" containerName="revoke-keys" Oct 30 21:56:33.818610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818459 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="015125ae-8075-4e8f-b53f-7e1a0062be96" containerName="revoke-keys" Oct 30 21:56:33.818610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818469 4926 cpu_manager.go:414] "containerMap: removing container" podUID="94e1f5fe-1324-4394-956d-8e315c4c3151" containerName="revoke-keys" Oct 30 21:56:33.818610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818474 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e1f5fe-1324-4394-956d-8e315c4c3151" containerName="revoke-keys" Oct 30 21:56:33.818610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818482 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d3117a72-cb57-4508-ac1d-e69000eb7da5" containerName="revoke-keys" Oct 30 21:56:33.818610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818487 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3117a72-cb57-4508-ac1d-e69000eb7da5" containerName="revoke-keys" Oct 30 21:56:33.818610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818495 4926 cpu_manager.go:414] "containerMap: removing container" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" Oct 30 21:56:33.818610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818501 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" Oct 30 21:56:33.818610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818513 4926 cpu_manager.go:414] "containerMap: removing container" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" Oct 30 21:56:33.818610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818517 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" Oct 30 21:56:33.818610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818525 4926 cpu_manager.go:414] "containerMap: removing container" podUID="c88fb565-e4b8-4389-852e-68b980559848" containerName="revoke-keys" Oct 30 21:56:33.818610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818530 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c88fb565-e4b8-4389-852e-68b980559848" containerName="revoke-keys" Oct 30 21:56:33.818610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818610 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc449fbe-a012-44a2-b57b-5ab7bbe1f44f" containerName="main" Oct 30 21:56:33.818963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818617 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="c88fb565-e4b8-4389-852e-68b980559848" containerName="revoke-keys" Oct 30 21:56:33.818963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818626 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" containerName="istio-proxy" Oct 30 21:56:33.818963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818631 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b30160-e47f-45ea-8d4c-c6450c1e86bd" containerName="main" Oct 30 21:56:33.818963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818639 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="015125ae-8075-4e8f-b53f-7e1a0062be96" containerName="revoke-keys" Oct 30 21:56:33.818963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818646 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" containerName="istio-proxy" Oct 30 21:56:33.818963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818652 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e1f5fe-1324-4394-956d-8e315c4c3151" containerName="revoke-keys" Oct 30 21:56:33.818963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818657 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3117a72-cb57-4508-ac1d-e69000eb7da5" containerName="revoke-keys" Oct 30 21:56:33.818963 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.818663 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1f7f668-8bb1-435b-a2d8-7340fc3b7c32" containerName="main" Oct 30 21:56:33.823762 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.823742 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9" Oct 30 21:56:33.842840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.842810 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9"] Oct 30 21:56:33.848878 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.848822 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:56:33.849012 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.848932 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:56:33.851752 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.851722 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:56:33.851873 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.851771 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:56:33.965612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.965573 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e7d1e4-bcfc-4135-aa6c-f996c4425c5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9\" (UID: \"28e7d1e4-bcfc-4135-aa6c-f996c4425c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9" Oct 30 21:56:33.965612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.965615 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nst4\" (UniqueName: \"kubernetes.io/projected/28e7d1e4-bcfc-4135-aa6c-f996c4425c5e-kube-api-access-8nst4\") pod \"maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9\" (UID: \"28e7d1e4-bcfc-4135-aa6c-f996c4425c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9" Oct 30 21:56:33.965873 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:33.965754 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e7d1e4-bcfc-4135-aa6c-f996c4425c5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9\" (UID: \"28e7d1e4-bcfc-4135-aa6c-f996c4425c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9" Oct 30 21:56:34.066976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:34.066936 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e7d1e4-bcfc-4135-aa6c-f996c4425c5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9\" (UID: \"28e7d1e4-bcfc-4135-aa6c-f996c4425c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9" Oct 30 21:56:34.066976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:34.066981 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nst4\" (UniqueName: \"kubernetes.io/projected/28e7d1e4-bcfc-4135-aa6c-f996c4425c5e-kube-api-access-8nst4\") pod \"maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9\" (UID: \"28e7d1e4-bcfc-4135-aa6c-f996c4425c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9" Oct 30 21:56:34.067415 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:34.067125 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e7d1e4-bcfc-4135-aa6c-f996c4425c5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9\" (UID: \"28e7d1e4-bcfc-4135-aa6c-f996c4425c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9" Oct 30 21:56:34.067796 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:34.067774 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e7d1e4-bcfc-4135-aa6c-f996c4425c5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9\" (UID: \"28e7d1e4-bcfc-4135-aa6c-f996c4425c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9" Oct 30 21:56:34.069462 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:34.069408 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e7d1e4-bcfc-4135-aa6c-f996c4425c5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9\" (UID: \"28e7d1e4-bcfc-4135-aa6c-f996c4425c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9" Oct 30 21:56:34.090674 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:34.090648 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nst4\" (UniqueName: \"kubernetes.io/projected/28e7d1e4-bcfc-4135-aa6c-f996c4425c5e-kube-api-access-8nst4\") pod \"maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9\" (UID: \"28e7d1e4-bcfc-4135-aa6c-f996c4425c5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9" Oct 30 21:56:34.138553 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:34.138515 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9" Oct 30 21:56:34.449793 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:34.449763 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9"] Oct 30 21:56:34.452117 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:56:34.452086 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e7d1e4_bcfc_4135_aa6c_f996c4425c5e.slice/crio-7c61d2d6e78434c73a3f4e1060f4aaa71090c16eb8e42237b364fc31a55d94af WatchSource:0}: Error finding container 7c61d2d6e78434c73a3f4e1060f4aaa71090c16eb8e42237b364fc31a55d94af: Status 404 returned error can't find the container with id 7c61d2d6e78434c73a3f4e1060f4aaa71090c16eb8e42237b364fc31a55d94af Oct 30 21:56:34.563977 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:34.563930 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9" event={"ID":"28e7d1e4-bcfc-4135-aa6c-f996c4425c5e","Type":"ContainerStarted","Data":"7c61d2d6e78434c73a3f4e1060f4aaa71090c16eb8e42237b364fc31a55d94af"} Oct 30 21:56:35.576725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:35.576619 4926 generic.go:356] "Generic (PLEG): container finished" podID="28e7d1e4-bcfc-4135-aa6c-f996c4425c5e" containerID="ff71d4878b12098011cd48588363343149659ca75ad3e4d51190b6a66ed1f074" exitCode=0 Oct 30 21:56:35.576725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:35.576702 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9" event={"ID":"28e7d1e4-bcfc-4135-aa6c-f996c4425c5e","Type":"ContainerDied","Data":"ff71d4878b12098011cd48588363343149659ca75ad3e4d51190b6a66ed1f074"} Oct 30 21:56:36.747142 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:36.747115 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9" Oct 30 21:56:36.901910 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:36.901819 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e7d1e4-bcfc-4135-aa6c-f996c4425c5e-kube-api-access-8nst4\" (UniqueName: \"kubernetes.io/projected/28e7d1e4-bcfc-4135-aa6c-f996c4425c5e-kube-api-access-8nst4\") pod \"28e7d1e4-bcfc-4135-aa6c-f996c4425c5e\" (UID: \"28e7d1e4-bcfc-4135-aa6c-f996c4425c5e\") " Oct 30 21:56:36.902092 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:36.902053 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28e7d1e4-bcfc-4135-aa6c-f996c4425c5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e7d1e4-bcfc-4135-aa6c-f996c4425c5e-maas-api-openshift-ca\") pod \"28e7d1e4-bcfc-4135-aa6c-f996c4425c5e\" (UID: \"28e7d1e4-bcfc-4135-aa6c-f996c4425c5e\") " Oct 30 21:56:36.902151 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:36.902098 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28e7d1e4-bcfc-4135-aa6c-f996c4425c5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e7d1e4-bcfc-4135-aa6c-f996c4425c5e-maas-api-xks-ca\") pod \"28e7d1e4-bcfc-4135-aa6c-f996c4425c5e\" (UID: \"28e7d1e4-bcfc-4135-aa6c-f996c4425c5e\") " Oct 30 21:56:36.902481 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:36.902449 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28e7d1e4-bcfc-4135-aa6c-f996c4425c5e-maas-api-openshift-ca" pod "28e7d1e4-bcfc-4135-aa6c-f996c4425c5e" (UID: "28e7d1e4-bcfc-4135-aa6c-f996c4425c5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:56:36.902640 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:36.902593 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28e7d1e4-bcfc-4135-aa6c-f996c4425c5e-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:36.904301 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:36.904274 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e7d1e4-bcfc-4135-aa6c-f996c4425c5e-kube-api-access-8nst4" pod "28e7d1e4-bcfc-4135-aa6c-f996c4425c5e" (UID: "28e7d1e4-bcfc-4135-aa6c-f996c4425c5e"). InnerVolumeSpecName "kube-api-access-8nst4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:56:36.904588 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:36.904563 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28e7d1e4-bcfc-4135-aa6c-f996c4425c5e-maas-api-xks-ca" pod "28e7d1e4-bcfc-4135-aa6c-f996c4425c5e" (UID: "28e7d1e4-bcfc-4135-aa6c-f996c4425c5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:56:37.003771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:37.003725 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nst4\" (UniqueName: \"kubernetes.io/projected/28e7d1e4-bcfc-4135-aa6c-f996c4425c5e-kube-api-access-8nst4\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:37.003771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:37.003770 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28e7d1e4-bcfc-4135-aa6c-f996c4425c5e-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:56:37.485968 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:56:37.485922 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:56:37.596159 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:37.596122 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9" event={"ID":"28e7d1e4-bcfc-4135-aa6c-f996c4425c5e","Type":"ContainerDied","Data":"7c61d2d6e78434c73a3f4e1060f4aaa71090c16eb8e42237b364fc31a55d94af"} Oct 30 21:56:37.596381 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:37.596167 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9" Oct 30 21:56:37.596381 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:37.596179 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c61d2d6e78434c73a3f4e1060f4aaa71090c16eb8e42237b364fc31a55d94af" Oct 30 21:56:39.925832 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:39.925795 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd"] Oct 30 21:56:39.936847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:39.936799 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-56ac74-989dd"] Oct 30 21:56:41.016604 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:41.016571 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29e075f3-e59c-4bbd-85af-311b46703f80" path="/var/lib/kubelet/pods/29e075f3-e59c-4bbd-85af-311b46703f80/volumes" Oct 30 21:56:41.042540 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:41.042504 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e3e153-gs7p4"] Oct 30 21:56:41.052870 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:41.052834 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7e3e153-gs7p4"] Oct 30 21:56:43.017109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:43.017070 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3285ca61-cc2b-4c14-9113-e2aaafcc7e53" path="/var/lib/kubelet/pods/3285ca61-cc2b-4c14-9113-e2aaafcc7e53/volumes" Oct 30 21:56:46.184412 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:46.184369 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng"] Oct 30 21:56:46.195819 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:46.195780 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-58fea9-qpdng"] Oct 30 21:56:47.016507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:47.016469 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="170ffbae-db1d-4d75-bc42-6548ea5c74bc" path="/var/lib/kubelet/pods/170ffbae-db1d-4d75-bc42-6548ea5c74bc/volumes" Oct 30 21:56:48.918109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:48.918021 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9"] Oct 30 21:56:48.928799 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:48.928755 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-c46397-xvnh9"] Oct 30 21:56:49.016201 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:49.016155 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94e1f5fe-1324-4394-956d-8e315c4c3151" path="/var/lib/kubelet/pods/94e1f5fe-1324-4394-956d-8e315c4c3151/volumes" Oct 30 21:56:53.865094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:53.865017 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq"] Oct 30 21:56:53.876275 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:53.876235 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-55c5e0-xkmqq"] Oct 30 21:56:55.018246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:56:55.018202 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3117a72-cb57-4508-ac1d-e69000eb7da5" path="/var/lib/kubelet/pods/d3117a72-cb57-4508-ac1d-e69000eb7da5/volumes" Oct 30 21:57:01.755648 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:01.755607 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7"] Oct 30 21:57:01.756122 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:01.756106 4926 cpu_manager.go:414] "containerMap: removing container" podUID="79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" containerName="istio-proxy" Oct 30 21:57:01.756198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:01.756123 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ee4f20-47b0-4c7a-9ad1-ec970173c3ee" containerName="istio-proxy" Oct 30 21:57:01.756198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:01.756159 4926 cpu_manager.go:414] "containerMap: removing container" podUID="28e7d1e4-bcfc-4135-aa6c-f996c4425c5e" containerName="revoke-keys" Oct 30 21:57:01.756198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:01.756165 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e7d1e4-bcfc-4135-aa6c-f996c4425c5e" containerName="revoke-keys" Oct 30 21:57:01.756305 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:01.756251 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e7d1e4-bcfc-4135-aa6c-f996c4425c5e" containerName="revoke-keys" Oct 30 21:57:01.761339 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:01.761316 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7" Oct 30 21:57:01.783927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:01.783894 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:57:01.784668 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:01.784633 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7"] Oct 30 21:57:01.791541 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:01.791506 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:57:01.791709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:01.791602 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:57:01.791709 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:01.791696 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:57:01.833733 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:01.833684 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz8sm\" (UniqueName: \"kubernetes.io/projected/d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18-kube-api-access-wz8sm\") pod \"maas-api-revoke-keys-e2e-worker-w4-839163-9njp7\" (UID: \"d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7" Oct 30 21:57:01.833912 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:01.833846 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-839163-9njp7\" (UID: \"d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7" Oct 30 21:57:01.833912 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:01.833886 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-839163-9njp7\" (UID: \"d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7" Oct 30 21:57:01.934787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:01.934736 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz8sm\" (UniqueName: \"kubernetes.io/projected/d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18-kube-api-access-wz8sm\") pod \"maas-api-revoke-keys-e2e-worker-w4-839163-9njp7\" (UID: \"d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7" Oct 30 21:57:01.935003 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:01.934829 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-839163-9njp7\" (UID: \"d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7" Oct 30 21:57:01.935003 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:01.934852 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-839163-9njp7\" (UID: \"d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7" Oct 30 21:57:01.935736 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:01.935707 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-839163-9njp7\" (UID: \"d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7" Oct 30 21:57:01.937410 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:01.937387 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-839163-9njp7\" (UID: \"d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7" Oct 30 21:57:01.955560 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:01.955528 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz8sm\" (UniqueName: \"kubernetes.io/projected/d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18-kube-api-access-wz8sm\") pod \"maas-api-revoke-keys-e2e-worker-w4-839163-9njp7\" (UID: \"d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7" Oct 30 21:57:02.080333 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:02.080236 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7" Oct 30 21:57:02.408847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:02.408818 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7"] Oct 30 21:57:02.410996 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:57:02.410960 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ea34a7_cc8b_4e21_9aab_ac38fa8e2d18.slice/crio-da9146f64c141f22a794c88c194fd24ed781aa53b71f98e1f9ef7ee8bad7fbda WatchSource:0}: Error finding container da9146f64c141f22a794c88c194fd24ed781aa53b71f98e1f9ef7ee8bad7fbda: Status 404 returned error can't find the container with id da9146f64c141f22a794c88c194fd24ed781aa53b71f98e1f9ef7ee8bad7fbda Oct 30 21:57:02.828820 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:02.828772 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7" event={"ID":"d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18","Type":"ContainerStarted","Data":"102999b4aec62309d83df6b4bb8077bbb6de23025185e029bd25cd1be2b18b17"} Oct 30 21:57:02.829321 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:02.828878 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7" event={"ID":"d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18","Type":"ContainerStarted","Data":"da9146f64c141f22a794c88c194fd24ed781aa53b71f98e1f9ef7ee8bad7fbda"} Oct 30 21:57:02.873838 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:02.873768 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7" podStartSLOduration=1.873748545 podStartE2EDuration="1.873748545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:02.871099745 +0000 UTC m=+3409.992173491" watchObservedRunningTime="2026-10-30 21:57:02.873748545 +0000 UTC m=+3409.994822291" Oct 30 21:57:03.838392 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:03.838356 4926 generic.go:356] "Generic (PLEG): container finished" podID="d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18" containerID="102999b4aec62309d83df6b4bb8077bbb6de23025185e029bd25cd1be2b18b17" exitCode=0 Oct 30 21:57:03.838875 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:03.838438 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7" event={"ID":"d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18","Type":"ContainerDied","Data":"102999b4aec62309d83df6b4bb8077bbb6de23025185e029bd25cd1be2b18b17"} Oct 30 21:57:04.381870 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:04.381821 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7"] Oct 30 21:57:04.382488 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:04.382454 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" podUID="97575fd5-0609-4000-904d-ad4cf40f23ef" containerName="istio-proxy" containerID="cri-o://4d42041383fe6050cabee4fc7c005f08c26fa95694420868934e1020598e06d1" gracePeriod=30 Oct 30 21:57:05.026541 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.026509 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7" Oct 30 21:57:05.111498 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.111438 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9"] Oct 30 21:57:05.114837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.114804 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-be3ada-clgt9"] Oct 30 21:57:05.172356 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.172323 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18-kube-api-access-wz8sm\" (UniqueName: \"kubernetes.io/projected/d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18-kube-api-access-wz8sm\") pod \"d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18\" (UID: \"d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18\") " Oct 30 21:57:05.172580 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.172441 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18-maas-api-xks-ca\") pod \"d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18\" (UID: \"d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18\") " Oct 30 21:57:05.172580 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.172531 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18-maas-api-openshift-ca\") pod \"d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18\" (UID: \"d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18\") " Oct 30 21:57:05.174645 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.173369 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18-maas-api-openshift-ca" pod "d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18" (UID: "d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:05.183992 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.182782 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18-maas-api-xks-ca" pod "d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18" (UID: "d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:05.184440 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.184410 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18-kube-api-access-wz8sm" pod "d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18" (UID: "d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18"). InnerVolumeSpecName "kube-api-access-wz8sm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:05.274249 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.274208 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:05.274249 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.274246 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz8sm\" (UniqueName: \"kubernetes.io/projected/d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18-kube-api-access-wz8sm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:05.274497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.274261 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:05.277507 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.277477 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w"] Oct 30 21:57:05.277985 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.277972 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18" containerName="revoke-keys" Oct 30 21:57:05.278041 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.277986 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18" containerName="revoke-keys" Oct 30 21:57:05.278148 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.278136 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18" containerName="revoke-keys" Oct 30 21:57:05.281647 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.281628 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w" Oct 30 21:57:05.300067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.300010 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w"] Oct 30 21:57:05.375368 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.375327 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d8d1ce-693f-4751-b6fe-ad4774673512-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w\" (UID: \"a7d8d1ce-693f-4751-b6fe-ad4774673512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w" Oct 30 21:57:05.375578 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.375535 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br9jt\" (UniqueName: \"kubernetes.io/projected/a7d8d1ce-693f-4751-b6fe-ad4774673512-kube-api-access-br9jt\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w\" (UID: \"a7d8d1ce-693f-4751-b6fe-ad4774673512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w" Oct 30 21:57:05.375664 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.375646 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d8d1ce-693f-4751-b6fe-ad4774673512-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w\" (UID: \"a7d8d1ce-693f-4751-b6fe-ad4774673512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w" Oct 30 21:57:05.476767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.476728 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d8d1ce-693f-4751-b6fe-ad4774673512-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w\" (UID: \"a7d8d1ce-693f-4751-b6fe-ad4774673512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w" Oct 30 21:57:05.476978 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.476836 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d8d1ce-693f-4751-b6fe-ad4774673512-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w\" (UID: \"a7d8d1ce-693f-4751-b6fe-ad4774673512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w" Oct 30 21:57:05.476978 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.476944 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br9jt\" (UniqueName: \"kubernetes.io/projected/a7d8d1ce-693f-4751-b6fe-ad4774673512-kube-api-access-br9jt\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w\" (UID: \"a7d8d1ce-693f-4751-b6fe-ad4774673512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w" Oct 30 21:57:05.477627 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.477592 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d8d1ce-693f-4751-b6fe-ad4774673512-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w\" (UID: \"a7d8d1ce-693f-4751-b6fe-ad4774673512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w" Oct 30 21:57:05.479416 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.479382 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d8d1ce-693f-4751-b6fe-ad4774673512-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w\" (UID: \"a7d8d1ce-693f-4751-b6fe-ad4774673512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w" Oct 30 21:57:05.495812 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.495779 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br9jt\" (UniqueName: \"kubernetes.io/projected/a7d8d1ce-693f-4751-b6fe-ad4774673512-kube-api-access-br9jt\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w\" (UID: \"a7d8d1ce-693f-4751-b6fe-ad4774673512\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w" Oct 30 21:57:05.597390 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.597356 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w" Oct 30 21:57:05.887115 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.886975 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7" event={"ID":"d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18","Type":"ContainerDied","Data":"da9146f64c141f22a794c88c194fd24ed781aa53b71f98e1f9ef7ee8bad7fbda"} Oct 30 21:57:05.887115 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.886990 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7" Oct 30 21:57:05.887343 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.887116 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da9146f64c141f22a794c88c194fd24ed781aa53b71f98e1f9ef7ee8bad7fbda" Oct 30 21:57:05.903966 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:05.903939 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w"] Oct 30 21:57:05.906467 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:57:05.906433 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d8d1ce_693f_4751_b6fe_ad4774673512.slice/crio-05a56953a3f24857036930e514ef76946829f9135ed547457f235598533e018a WatchSource:0}: Error finding container 05a56953a3f24857036930e514ef76946829f9135ed547457f235598533e018a: Status 404 returned error can't find the container with id 05a56953a3f24857036930e514ef76946829f9135ed547457f235598533e018a Oct 30 21:57:06.516966 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:06.516876 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm"] Oct 30 21:57:06.898194 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:06.898158 4926 generic.go:356] "Generic (PLEG): container finished" podID="a7d8d1ce-693f-4751-b6fe-ad4774673512" containerID="1e377cb55c705bd6b7021bdb000934f274c87ca5142b873ce2b073a9a1e76b69" exitCode=0 Oct 30 21:57:06.898403 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:06.898221 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w" event={"ID":"a7d8d1ce-693f-4751-b6fe-ad4774673512","Type":"ContainerDied","Data":"1e377cb55c705bd6b7021bdb000934f274c87ca5142b873ce2b073a9a1e76b69"} Oct 30 21:57:06.898403 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:06.898261 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w" event={"ID":"a7d8d1ce-693f-4751-b6fe-ad4774673512","Type":"ContainerStarted","Data":"05a56953a3f24857036930e514ef76946829f9135ed547457f235598533e018a"} Oct 30 21:57:07.016402 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:07.016367 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28e7d1e4-bcfc-4135-aa6c-f996c4425c5e" path="/var/lib/kubelet/pods/28e7d1e4-bcfc-4135-aa6c-f996c4425c5e/volumes" Oct 30 21:57:07.487137 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:57:07.487097 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:57:08.061520 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:08.061492 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w" Oct 30 21:57:08.211781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:08.211738 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7d8d1ce-693f-4751-b6fe-ad4774673512-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d8d1ce-693f-4751-b6fe-ad4774673512-maas-api-xks-ca\") pod \"a7d8d1ce-693f-4751-b6fe-ad4774673512\" (UID: \"a7d8d1ce-693f-4751-b6fe-ad4774673512\") " Oct 30 21:57:08.211976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:08.211886 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7d8d1ce-693f-4751-b6fe-ad4774673512-kube-api-access-br9jt\" (UniqueName: \"kubernetes.io/projected/a7d8d1ce-693f-4751-b6fe-ad4774673512-kube-api-access-br9jt\") pod \"a7d8d1ce-693f-4751-b6fe-ad4774673512\" (UID: \"a7d8d1ce-693f-4751-b6fe-ad4774673512\") " Oct 30 21:57:08.211976 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:08.211967 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7d8d1ce-693f-4751-b6fe-ad4774673512-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d8d1ce-693f-4751-b6fe-ad4774673512-maas-api-openshift-ca\") pod \"a7d8d1ce-693f-4751-b6fe-ad4774673512\" (UID: \"a7d8d1ce-693f-4751-b6fe-ad4774673512\") " Oct 30 21:57:08.212368 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:08.212336 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7d8d1ce-693f-4751-b6fe-ad4774673512-maas-api-openshift-ca" pod "a7d8d1ce-693f-4751-b6fe-ad4774673512" (UID: "a7d8d1ce-693f-4751-b6fe-ad4774673512"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:08.212497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:08.212435 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7d8d1ce-693f-4751-b6fe-ad4774673512-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:08.214043 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:08.214010 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7d8d1ce-693f-4751-b6fe-ad4774673512-kube-api-access-br9jt" pod "a7d8d1ce-693f-4751-b6fe-ad4774673512" (UID: "a7d8d1ce-693f-4751-b6fe-ad4774673512"). InnerVolumeSpecName "kube-api-access-br9jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:08.214178 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:08.214089 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7d8d1ce-693f-4751-b6fe-ad4774673512-maas-api-xks-ca" pod "a7d8d1ce-693f-4751-b6fe-ad4774673512" (UID: "a7d8d1ce-693f-4751-b6fe-ad4774673512"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:08.313618 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:08.313585 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7d8d1ce-693f-4751-b6fe-ad4774673512-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:08.313618 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:08.313614 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br9jt\" (UniqueName: \"kubernetes.io/projected/a7d8d1ce-693f-4751-b6fe-ad4774673512-kube-api-access-br9jt\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:08.846732 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:08.846684 4926 patch_prober.go:29] interesting pod/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:57:08.846922 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:08.846752 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" podUID="97575fd5-0609-4000-904d-ad4cf40f23ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:57:08.918463 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:08.918432 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w" Oct 30 21:57:08.918463 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:08.918457 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w" event={"ID":"a7d8d1ce-693f-4751-b6fe-ad4774673512","Type":"ContainerDied","Data":"05a56953a3f24857036930e514ef76946829f9135ed547457f235598533e018a"} Oct 30 21:57:08.918744 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:08.918506 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05a56953a3f24857036930e514ef76946829f9135ed547457f235598533e018a" Oct 30 21:57:09.660736 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.660670 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:57:09.830394 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.830339 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-envoy\") pod \"97575fd5-0609-4000-904d-ad4cf40f23ef\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " Oct 30 21:57:09.830608 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.830446 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-workload-socket\") pod \"97575fd5-0609-4000-904d-ad4cf40f23ef\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " Oct 30 21:57:09.830608 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.830513 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-workload-certs\") pod \"97575fd5-0609-4000-904d-ad4cf40f23ef\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " Oct 30 21:57:09.830608 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.830554 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97575fd5-0609-4000-904d-ad4cf40f23ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97575fd5-0609-4000-904d-ad4cf40f23ef-istiod-ca-cert\") pod \"97575fd5-0609-4000-904d-ad4cf40f23ef\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " Oct 30 21:57:09.830608 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.830584 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-credential-socket\") pod \"97575fd5-0609-4000-904d-ad4cf40f23ef\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " Oct 30 21:57:09.830855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.830708 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-data\") pod \"97575fd5-0609-4000-904d-ad4cf40f23ef\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " Oct 30 21:57:09.830855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.830739 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-workload-socket" pod "97575fd5-0609-4000-904d-ad4cf40f23ef" (UID: "97575fd5-0609-4000-904d-ad4cf40f23ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:09.830855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.830795 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-podinfo\") pod \"97575fd5-0609-4000-904d-ad4cf40f23ef\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " Oct 30 21:57:09.830855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.830826 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97575fd5-0609-4000-904d-ad4cf40f23ef-kube-api-access-cnkfv\" (UniqueName: \"kubernetes.io/projected/97575fd5-0609-4000-904d-ad4cf40f23ef-kube-api-access-cnkfv\") pod \"97575fd5-0609-4000-904d-ad4cf40f23ef\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " Oct 30 21:57:09.831107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.830868 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-token\" (UniqueName: \"kubernetes.io/projected/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-token\") pod \"97575fd5-0609-4000-904d-ad4cf40f23ef\" (UID: \"97575fd5-0609-4000-904d-ad4cf40f23ef\") " Oct 30 21:57:09.831107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.830867 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-workload-certs" pod "97575fd5-0609-4000-904d-ad4cf40f23ef" (UID: "97575fd5-0609-4000-904d-ad4cf40f23ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:09.831107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.830883 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-credential-socket" pod "97575fd5-0609-4000-904d-ad4cf40f23ef" (UID: "97575fd5-0609-4000-904d-ad4cf40f23ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:09.831107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.831009 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-data" pod "97575fd5-0609-4000-904d-ad4cf40f23ef" (UID: "97575fd5-0609-4000-904d-ad4cf40f23ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:09.831107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.831045 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97575fd5-0609-4000-904d-ad4cf40f23ef-istiod-ca-cert" pod "97575fd5-0609-4000-904d-ad4cf40f23ef" (UID: "97575fd5-0609-4000-904d-ad4cf40f23ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:09.831407 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.831361 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:09.831483 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.831412 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97575fd5-0609-4000-904d-ad4cf40f23ef-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:09.831483 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.831427 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:09.831483 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.831441 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:09.831483 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.831454 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:09.833319 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.833289 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-envoy" pod "97575fd5-0609-4000-904d-ad4cf40f23ef" (UID: "97575fd5-0609-4000-904d-ad4cf40f23ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:09.833508 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.833486 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-token" pod "97575fd5-0609-4000-904d-ad4cf40f23ef" (UID: "97575fd5-0609-4000-904d-ad4cf40f23ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:09.833631 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.833508 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-podinfo" pod "97575fd5-0609-4000-904d-ad4cf40f23ef" (UID: "97575fd5-0609-4000-904d-ad4cf40f23ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:09.833703 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.833633 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97575fd5-0609-4000-904d-ad4cf40f23ef-kube-api-access-cnkfv" pod "97575fd5-0609-4000-904d-ad4cf40f23ef" (UID: "97575fd5-0609-4000-904d-ad4cf40f23ef"). InnerVolumeSpecName "kube-api-access-cnkfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:09.927987 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.927949 4926 generic.go:356] "Generic (PLEG): container finished" podID="97575fd5-0609-4000-904d-ad4cf40f23ef" containerID="4d42041383fe6050cabee4fc7c005f08c26fa95694420868934e1020598e06d1" exitCode=0 Oct 30 21:57:09.928256 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.928006 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" Oct 30 21:57:09.928256 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.928042 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" event={"ID":"97575fd5-0609-4000-904d-ad4cf40f23ef","Type":"ContainerDied","Data":"4d42041383fe6050cabee4fc7c005f08c26fa95694420868934e1020598e06d1"} Oct 30 21:57:09.928256 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.928101 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7" event={"ID":"97575fd5-0609-4000-904d-ad4cf40f23ef","Type":"ContainerDied","Data":"e0ec45cf4e9cad6f77d71f2e5ff7f5e1d9913d8f0972df5e54a3feba0a276ed1"} Oct 30 21:57:09.928256 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.928130 4926 scope.go:122] "RemoveContainer" containerID="4d42041383fe6050cabee4fc7c005f08c26fa95694420868934e1020598e06d1" Oct 30 21:57:09.932363 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.932335 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:09.932497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.932367 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:09.932497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.932382 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnkfv\" (UniqueName: \"kubernetes.io/projected/97575fd5-0609-4000-904d-ad4cf40f23ef-kube-api-access-cnkfv\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:09.932497 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.932397 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97575fd5-0609-4000-904d-ad4cf40f23ef-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:09.944804 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.943749 4926 scope.go:122] "RemoveContainer" containerID="4d42041383fe6050cabee4fc7c005f08c26fa95694420868934e1020598e06d1" Oct 30 21:57:09.945777 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:57:09.945015 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d42041383fe6050cabee4fc7c005f08c26fa95694420868934e1020598e06d1\": container with ID starting with 4d42041383fe6050cabee4fc7c005f08c26fa95694420868934e1020598e06d1 not found: ID does not exist" containerID="4d42041383fe6050cabee4fc7c005f08c26fa95694420868934e1020598e06d1" Oct 30 21:57:09.945903 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.945791 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d42041383fe6050cabee4fc7c005f08c26fa95694420868934e1020598e06d1"} err="failed to get container status \"4d42041383fe6050cabee4fc7c005f08c26fa95694420868934e1020598e06d1\": rpc error: code = NotFound desc = could not find container \"4d42041383fe6050cabee4fc7c005f08c26fa95694420868934e1020598e06d1\": container with ID starting with 4d42041383fe6050cabee4fc7c005f08c26fa95694420868934e1020598e06d1 not found: ID does not exist" Oct 30 21:57:09.983309 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.983271 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7"] Oct 30 21:57:09.990308 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:09.990274 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-55c5e0-openshift-default-85bfdf9f8c-f4nj7"] Oct 30 21:57:11.015525 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:11.015488 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97575fd5-0609-4000-904d-ad4cf40f23ef" path="/var/lib/kubelet/pods/97575fd5-0609-4000-904d-ad4cf40f23ef/volumes" Oct 30 21:57:12.174960 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:12.174924 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92"] Oct 30 21:57:12.206568 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:12.206524 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft"] Oct 30 21:57:13.868582 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:13.868517 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45"] Oct 30 21:57:13.873187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:13.872958 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" podUID="e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" containerName="istio-proxy" containerID="cri-o://6556f37eef023f1c2c84e045cab721271550a5488205f608f502a2bb40dd1574" gracePeriod=30 Oct 30 21:57:14.384872 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.384836 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7"] Oct 30 21:57:14.391049 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.390991 4926 cpu_manager.go:414] "containerMap: removing container" podUID="97575fd5-0609-4000-904d-ad4cf40f23ef" containerName="istio-proxy" Oct 30 21:57:14.391049 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.391047 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97575fd5-0609-4000-904d-ad4cf40f23ef" containerName="istio-proxy" Oct 30 21:57:14.391301 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.391131 4926 cpu_manager.go:414] "containerMap: removing container" podUID="a7d8d1ce-693f-4751-b6fe-ad4774673512" containerName="revoke-keys" Oct 30 21:57:14.391301 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.391140 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7d8d1ce-693f-4751-b6fe-ad4774673512" containerName="revoke-keys" Oct 30 21:57:14.391448 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.391432 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="97575fd5-0609-4000-904d-ad4cf40f23ef" containerName="istio-proxy" Oct 30 21:57:14.391515 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.391459 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7d8d1ce-693f-4751-b6fe-ad4774673512" containerName="revoke-keys" Oct 30 21:57:14.403049 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.403008 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7" Oct 30 21:57:14.419331 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.419220 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:57:14.419331 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.419282 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:57:14.419838 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.419808 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7"] Oct 30 21:57:14.420203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.420181 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:57:14.420309 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.420207 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:57:14.484269 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.484220 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttrnx\" (UniqueName: \"kubernetes.io/projected/25313085-efd3-4732-bd01-91c3ee701a4b-kube-api-access-ttrnx\") pod \"maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7\" (UID: \"25313085-efd3-4732-bd01-91c3ee701a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7" Oct 30 21:57:14.484493 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.484382 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25313085-efd3-4732-bd01-91c3ee701a4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7\" (UID: \"25313085-efd3-4732-bd01-91c3ee701a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7" Oct 30 21:57:14.484493 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.484432 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25313085-efd3-4732-bd01-91c3ee701a4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7\" (UID: \"25313085-efd3-4732-bd01-91c3ee701a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7" Oct 30 21:57:14.585107 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.585062 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25313085-efd3-4732-bd01-91c3ee701a4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7\" (UID: \"25313085-efd3-4732-bd01-91c3ee701a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7" Oct 30 21:57:14.585312 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.585137 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25313085-efd3-4732-bd01-91c3ee701a4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7\" (UID: \"25313085-efd3-4732-bd01-91c3ee701a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7" Oct 30 21:57:14.585312 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.585212 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttrnx\" (UniqueName: \"kubernetes.io/projected/25313085-efd3-4732-bd01-91c3ee701a4b-kube-api-access-ttrnx\") pod \"maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7\" (UID: \"25313085-efd3-4732-bd01-91c3ee701a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7" Oct 30 21:57:14.585896 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.585868 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25313085-efd3-4732-bd01-91c3ee701a4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7\" (UID: \"25313085-efd3-4732-bd01-91c3ee701a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7" Oct 30 21:57:14.587673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.587646 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25313085-efd3-4732-bd01-91c3ee701a4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7\" (UID: \"25313085-efd3-4732-bd01-91c3ee701a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7" Oct 30 21:57:14.603922 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.603890 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttrnx\" (UniqueName: \"kubernetes.io/projected/25313085-efd3-4732-bd01-91c3ee701a4b-kube-api-access-ttrnx\") pod \"maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7\" (UID: \"25313085-efd3-4732-bd01-91c3ee701a4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7" Oct 30 21:57:14.722140 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:14.722100 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7" Oct 30 21:57:15.032815 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:15.032780 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7"] Oct 30 21:57:15.033853 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:57:15.033823 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25313085_efd3_4732_bd01_91c3ee701a4b.slice/crio-54798133e3704ec612ff729a383c7033df8a5aece3e08b9f5b19c6282bfbb61f WatchSource:0}: Error finding container 54798133e3704ec612ff729a383c7033df8a5aece3e08b9f5b19c6282bfbb61f: Status 404 returned error can't find the container with id 54798133e3704ec612ff729a383c7033df8a5aece3e08b9f5b19c6282bfbb61f Oct 30 21:57:15.979390 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:15.979347 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7" event={"ID":"25313085-efd3-4732-bd01-91c3ee701a4b","Type":"ContainerStarted","Data":"3f235a3876f505c8bf9caed20e205281bd2408b1d729252af13afc5d8555087b"} Oct 30 21:57:15.979621 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:15.979416 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7" event={"ID":"25313085-efd3-4732-bd01-91c3ee701a4b","Type":"ContainerStarted","Data":"54798133e3704ec612ff729a383c7033df8a5aece3e08b9f5b19c6282bfbb61f"} Oct 30 21:57:15.997267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:15.997236 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c54e7e6b-4l64w"] Oct 30 21:57:16.001106 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.001084 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54e7e6b-4l64w" Oct 30 21:57:16.017439 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.017409 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg"] Oct 30 21:57:16.021297 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.021278 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg" Oct 30 21:57:16.026610 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.026561 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7" podStartSLOduration=2.026548297 podStartE2EDuration="2.026548297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:16.024713308 +0000 UTC m=+3423.145787054" watchObservedRunningTime="2026-10-30 21:57:16.026548297 +0000 UTC m=+3423.147622042" Oct 30 21:57:16.040004 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.039968 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c54e7e6b-4l64w"] Oct 30 21:57:16.054590 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.054544 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg"] Oct 30 21:57:16.106738 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.106689 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cae23f38-df9f-443a-bf6b-aa20e5e365cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c54e7e6b-4l64w\" (UID: \"cae23f38-df9f-443a-bf6b-aa20e5e365cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54e7e6b-4l64w" Oct 30 21:57:16.106918 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.106840 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c268\" (UniqueName: \"kubernetes.io/projected/e89d855c-7de9-409a-9644-474afb490832-kube-api-access-2c268\") pod \"maas-api-revoke-sub-9d5f588f-2pvcg\" (UID: \"e89d855c-7de9-409a-9644-474afb490832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg" Oct 30 21:57:16.107001 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.106965 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89d855c-7de9-409a-9644-474afb490832-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d5f588f-2pvcg\" (UID: \"e89d855c-7de9-409a-9644-474afb490832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg" Oct 30 21:57:16.107130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.107114 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cae23f38-df9f-443a-bf6b-aa20e5e365cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c54e7e6b-4l64w\" (UID: \"cae23f38-df9f-443a-bf6b-aa20e5e365cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54e7e6b-4l64w" Oct 30 21:57:16.107175 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.107160 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spdsh\" (UniqueName: \"kubernetes.io/projected/cae23f38-df9f-443a-bf6b-aa20e5e365cd-kube-api-access-spdsh\") pod \"maas-api-revoke-sub-c54e7e6b-4l64w\" (UID: \"cae23f38-df9f-443a-bf6b-aa20e5e365cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54e7e6b-4l64w" Oct 30 21:57:16.107255 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.107236 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89d855c-7de9-409a-9644-474afb490832-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d5f588f-2pvcg\" (UID: \"e89d855c-7de9-409a-9644-474afb490832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg" Oct 30 21:57:16.208850 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.208804 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cae23f38-df9f-443a-bf6b-aa20e5e365cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c54e7e6b-4l64w\" (UID: \"cae23f38-df9f-443a-bf6b-aa20e5e365cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54e7e6b-4l64w" Oct 30 21:57:16.209086 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.208875 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spdsh\" (UniqueName: \"kubernetes.io/projected/cae23f38-df9f-443a-bf6b-aa20e5e365cd-kube-api-access-spdsh\") pod \"maas-api-revoke-sub-c54e7e6b-4l64w\" (UID: \"cae23f38-df9f-443a-bf6b-aa20e5e365cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54e7e6b-4l64w" Oct 30 21:57:16.209086 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.208903 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89d855c-7de9-409a-9644-474afb490832-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d5f588f-2pvcg\" (UID: \"e89d855c-7de9-409a-9644-474afb490832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg" Oct 30 21:57:16.209086 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.208946 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cae23f38-df9f-443a-bf6b-aa20e5e365cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c54e7e6b-4l64w\" (UID: \"cae23f38-df9f-443a-bf6b-aa20e5e365cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54e7e6b-4l64w" Oct 30 21:57:16.209086 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.209068 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c268\" (UniqueName: \"kubernetes.io/projected/e89d855c-7de9-409a-9644-474afb490832-kube-api-access-2c268\") pod \"maas-api-revoke-sub-9d5f588f-2pvcg\" (UID: \"e89d855c-7de9-409a-9644-474afb490832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg" Oct 30 21:57:16.209341 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.209292 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89d855c-7de9-409a-9644-474afb490832-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d5f588f-2pvcg\" (UID: \"e89d855c-7de9-409a-9644-474afb490832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg" Oct 30 21:57:16.210237 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.210199 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89d855c-7de9-409a-9644-474afb490832-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d5f588f-2pvcg\" (UID: \"e89d855c-7de9-409a-9644-474afb490832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg" Oct 30 21:57:16.210387 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.210291 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cae23f38-df9f-443a-bf6b-aa20e5e365cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c54e7e6b-4l64w\" (UID: \"cae23f38-df9f-443a-bf6b-aa20e5e365cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54e7e6b-4l64w" Oct 30 21:57:16.212118 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.212094 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89d855c-7de9-409a-9644-474afb490832-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d5f588f-2pvcg\" (UID: \"e89d855c-7de9-409a-9644-474afb490832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg" Oct 30 21:57:16.212224 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.212126 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cae23f38-df9f-443a-bf6b-aa20e5e365cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c54e7e6b-4l64w\" (UID: \"cae23f38-df9f-443a-bf6b-aa20e5e365cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54e7e6b-4l64w" Oct 30 21:57:16.236841 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.236770 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c268\" (UniqueName: \"kubernetes.io/projected/e89d855c-7de9-409a-9644-474afb490832-kube-api-access-2c268\") pod \"maas-api-revoke-sub-9d5f588f-2pvcg\" (UID: \"e89d855c-7de9-409a-9644-474afb490832\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg" Oct 30 21:57:16.236841 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.236803 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spdsh\" (UniqueName: \"kubernetes.io/projected/cae23f38-df9f-443a-bf6b-aa20e5e365cd-kube-api-access-spdsh\") pod \"maas-api-revoke-sub-c54e7e6b-4l64w\" (UID: \"cae23f38-df9f-443a-bf6b-aa20e5e365cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54e7e6b-4l64w" Oct 30 21:57:16.376573 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.376538 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54e7e6b-4l64w" Oct 30 21:57:16.388146 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.388109 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg" Oct 30 21:57:16.685472 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.685439 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c54e7e6b-4l64w"] Oct 30 21:57:16.686667 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:57:16.686635 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae23f38_df9f_443a_bf6b_aa20e5e365cd.slice/crio-a3006ef0c462269206296f0284b44e63556444f863a6222fc6626a000abb58df WatchSource:0}: Error finding container a3006ef0c462269206296f0284b44e63556444f863a6222fc6626a000abb58df: Status 404 returned error can't find the container with id a3006ef0c462269206296f0284b44e63556444f863a6222fc6626a000abb58df Oct 30 21:57:16.710219 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.710130 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg"] Oct 30 21:57:16.880539 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.880508 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:57:16.989623 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.989582 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54e7e6b-4l64w" event={"ID":"cae23f38-df9f-443a-bf6b-aa20e5e365cd","Type":"ContainerStarted","Data":"a3006ef0c462269206296f0284b44e63556444f863a6222fc6626a000abb58df"} Oct 30 21:57:16.991924 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.991896 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg" event={"ID":"e89d855c-7de9-409a-9644-474afb490832","Type":"ContainerStarted","Data":"d83b4be314076531ebfc8bd2f28f85853b2e4264d7989c70f4999b3e83c058d4"} Oct 30 21:57:16.992070 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.991948 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg" event={"ID":"e89d855c-7de9-409a-9644-474afb490832","Type":"ContainerStarted","Data":"b624f33e33efa5f611b92ee488e47184506f2279d1113a7fc575a381472ef24e"} Oct 30 21:57:16.993945 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.993918 4926 generic.go:356] "Generic (PLEG): container finished" podID="25313085-efd3-4732-bd01-91c3ee701a4b" containerID="3f235a3876f505c8bf9caed20e205281bd2408b1d729252af13afc5d8555087b" exitCode=0 Oct 30 21:57:16.994072 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:16.993998 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7" event={"ID":"25313085-efd3-4732-bd01-91c3ee701a4b","Type":"ContainerDied","Data":"3f235a3876f505c8bf9caed20e205281bd2408b1d729252af13afc5d8555087b"} Oct 30 21:57:17.034626 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:17.034519 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg" podStartSLOduration=2.034506691 podStartE2EDuration="2.034506691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:17.032838658 +0000 UTC m=+3424.153912406" watchObservedRunningTime="2026-10-30 21:57:17.034506691 +0000 UTC m=+3424.155580437" Oct 30 21:57:17.175575 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:17.175532 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92" podUID="586f88e9-d006-46ea-bef3-68cc117aaa81" containerName="payload-processing" containerID="cri-o://1428cc18e61ee2b52e2b0c2de322a8bcd85295aa31c4dda54b09bbc5de0295b1" gracePeriod=25 Oct 30 21:57:17.207168 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:17.207114 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft" podUID="11be824c-778f-4301-b2c5-7b0c8ca3d04c" containerName="payload-pre-processing" containerID="cri-o://42e68838bfebe24dff828eb4163541fa2404849a78b58a906e0f21ac908e3df0" gracePeriod=25 Oct 30 21:57:17.427612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:17.427583 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92" Oct 30 21:57:17.522163 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:17.522122 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/586f88e9-d006-46ea-bef3-68cc117aaa81-kube-api-access-cwtpc\" (UniqueName: \"kubernetes.io/projected/586f88e9-d006-46ea-bef3-68cc117aaa81-kube-api-access-cwtpc\") pod \"586f88e9-d006-46ea-bef3-68cc117aaa81\" (UID: \"586f88e9-d006-46ea-bef3-68cc117aaa81\") " Oct 30 21:57:17.522376 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:17.522217 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/586f88e9-d006-46ea-bef3-68cc117aaa81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/586f88e9-d006-46ea-bef3-68cc117aaa81-plugins-config-volume\") pod \"586f88e9-d006-46ea-bef3-68cc117aaa81\" (UID: \"586f88e9-d006-46ea-bef3-68cc117aaa81\") " Oct 30 21:57:17.522654 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:17.522629 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/586f88e9-d006-46ea-bef3-68cc117aaa81-plugins-config-volume" pod "586f88e9-d006-46ea-bef3-68cc117aaa81" (UID: "586f88e9-d006-46ea-bef3-68cc117aaa81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:17.524288 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:17.524259 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/586f88e9-d006-46ea-bef3-68cc117aaa81-kube-api-access-cwtpc" pod "586f88e9-d006-46ea-bef3-68cc117aaa81" (UID: "586f88e9-d006-46ea-bef3-68cc117aaa81"). InnerVolumeSpecName "kube-api-access-cwtpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:17.623992 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:17.623961 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwtpc\" (UniqueName: \"kubernetes.io/projected/586f88e9-d006-46ea-bef3-68cc117aaa81-kube-api-access-cwtpc\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:17.623992 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:17.623992 4926 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/586f88e9-d006-46ea-bef3-68cc117aaa81-plugins-config-volume\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:18.003448 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.003412 4926 generic.go:356] "Generic (PLEG): container finished" podID="e89d855c-7de9-409a-9644-474afb490832" containerID="d83b4be314076531ebfc8bd2f28f85853b2e4264d7989c70f4999b3e83c058d4" exitCode=0 Oct 30 21:57:18.003629 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.003491 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg" event={"ID":"e89d855c-7de9-409a-9644-474afb490832","Type":"ContainerDied","Data":"d83b4be314076531ebfc8bd2f28f85853b2e4264d7989c70f4999b3e83c058d4"} Oct 30 21:57:18.008439 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.008404 4926 generic.go:356] "Generic (PLEG): container finished" podID="586f88e9-d006-46ea-bef3-68cc117aaa81" containerID="1428cc18e61ee2b52e2b0c2de322a8bcd85295aa31c4dda54b09bbc5de0295b1" exitCode=0 Oct 30 21:57:18.008892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.008868 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92" event={"ID":"586f88e9-d006-46ea-bef3-68cc117aaa81","Type":"ContainerDied","Data":"1428cc18e61ee2b52e2b0c2de322a8bcd85295aa31c4dda54b09bbc5de0295b1"} Oct 30 21:57:18.009117 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.009097 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92" event={"ID":"586f88e9-d006-46ea-bef3-68cc117aaa81","Type":"ContainerDied","Data":"8ecc39eec06a3e28f33daeaf601a6cb554d66f8d1a56f53f3c9a5938f771bf24"} Oct 30 21:57:18.009274 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.009260 4926 scope.go:122] "RemoveContainer" containerID="1428cc18e61ee2b52e2b0c2de322a8bcd85295aa31c4dda54b09bbc5de0295b1" Oct 30 21:57:18.009586 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.009555 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92" Oct 30 21:57:18.017326 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.017300 4926 generic.go:356] "Generic (PLEG): container finished" podID="cae23f38-df9f-443a-bf6b-aa20e5e365cd" containerID="0760f5cc38f4f34116249ec423418145932d2d4d7cc531420af75dcc7e6a0621" exitCode=0 Oct 30 21:57:18.017573 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.017521 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54e7e6b-4l64w" event={"ID":"cae23f38-df9f-443a-bf6b-aa20e5e365cd","Type":"ContainerDied","Data":"0760f5cc38f4f34116249ec423418145932d2d4d7cc531420af75dcc7e6a0621"} Oct 30 21:57:18.028082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.028058 4926 scope.go:122] "RemoveContainer" containerID="1428cc18e61ee2b52e2b0c2de322a8bcd85295aa31c4dda54b09bbc5de0295b1" Oct 30 21:57:18.028381 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:57:18.028356 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1428cc18e61ee2b52e2b0c2de322a8bcd85295aa31c4dda54b09bbc5de0295b1\": container with ID starting with 1428cc18e61ee2b52e2b0c2de322a8bcd85295aa31c4dda54b09bbc5de0295b1 not found: ID does not exist" containerID="1428cc18e61ee2b52e2b0c2de322a8bcd85295aa31c4dda54b09bbc5de0295b1" Oct 30 21:57:18.028447 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.028394 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1428cc18e61ee2b52e2b0c2de322a8bcd85295aa31c4dda54b09bbc5de0295b1"} err="failed to get container status \"1428cc18e61ee2b52e2b0c2de322a8bcd85295aa31c4dda54b09bbc5de0295b1\": rpc error: code = NotFound desc = could not find container \"1428cc18e61ee2b52e2b0c2de322a8bcd85295aa31c4dda54b09bbc5de0295b1\": container with ID starting with 1428cc18e61ee2b52e2b0c2de322a8bcd85295aa31c4dda54b09bbc5de0295b1 not found: ID does not exist" Oct 30 21:57:18.132323 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.132288 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92"] Oct 30 21:57:18.140315 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.140279 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-839163-5bf99f9954-qlx92"] Oct 30 21:57:18.190197 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.190166 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7" Oct 30 21:57:18.231182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.231140 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25313085-efd3-4732-bd01-91c3ee701a4b-kube-api-access-ttrnx\" (UniqueName: \"kubernetes.io/projected/25313085-efd3-4732-bd01-91c3ee701a4b-kube-api-access-ttrnx\") pod \"25313085-efd3-4732-bd01-91c3ee701a4b\" (UID: \"25313085-efd3-4732-bd01-91c3ee701a4b\") " Oct 30 21:57:18.231409 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.231336 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25313085-efd3-4732-bd01-91c3ee701a4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25313085-efd3-4732-bd01-91c3ee701a4b-maas-api-openshift-ca\") pod \"25313085-efd3-4732-bd01-91c3ee701a4b\" (UID: \"25313085-efd3-4732-bd01-91c3ee701a4b\") " Oct 30 21:57:18.231409 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.231388 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25313085-efd3-4732-bd01-91c3ee701a4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25313085-efd3-4732-bd01-91c3ee701a4b-maas-api-xks-ca\") pod \"25313085-efd3-4732-bd01-91c3ee701a4b\" (UID: \"25313085-efd3-4732-bd01-91c3ee701a4b\") " Oct 30 21:57:18.231925 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.231895 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25313085-efd3-4732-bd01-91c3ee701a4b-maas-api-openshift-ca" pod "25313085-efd3-4732-bd01-91c3ee701a4b" (UID: "25313085-efd3-4732-bd01-91c3ee701a4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:18.233554 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.233528 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25313085-efd3-4732-bd01-91c3ee701a4b-kube-api-access-ttrnx" pod "25313085-efd3-4732-bd01-91c3ee701a4b" (UID: "25313085-efd3-4732-bd01-91c3ee701a4b"). InnerVolumeSpecName "kube-api-access-ttrnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:18.233726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.233708 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25313085-efd3-4732-bd01-91c3ee701a4b-maas-api-xks-ca" pod "25313085-efd3-4732-bd01-91c3ee701a4b" (UID: "25313085-efd3-4732-bd01-91c3ee701a4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:18.332652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.332544 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25313085-efd3-4732-bd01-91c3ee701a4b-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:18.332652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.332583 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25313085-efd3-4732-bd01-91c3ee701a4b-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:18.332652 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:18.332598 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttrnx\" (UniqueName: \"kubernetes.io/projected/25313085-efd3-4732-bd01-91c3ee701a4b-kube-api-access-ttrnx\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:19.016326 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.016294 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="586f88e9-d006-46ea-bef3-68cc117aaa81" path="/var/lib/kubelet/pods/586f88e9-d006-46ea-bef3-68cc117aaa81/volumes" Oct 30 21:57:19.028801 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.028772 4926 generic.go:356] "Generic (PLEG): container finished" podID="e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" containerID="6556f37eef023f1c2c84e045cab721271550a5488205f608f502a2bb40dd1574" exitCode=0 Oct 30 21:57:19.028923 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.028843 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" event={"ID":"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6","Type":"ContainerDied","Data":"6556f37eef023f1c2c84e045cab721271550a5488205f608f502a2bb40dd1574"} Oct 30 21:57:19.028923 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.028918 4926 scope.go:122] "RemoveContainer" containerID="2068931c23566279d1cb1eb15b62341ab40c38c9d3d0c4d36f7aefd8643d0b06" Oct 30 21:57:19.030499 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.030476 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7" Oct 30 21:57:19.030605 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.030490 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7" event={"ID":"25313085-efd3-4732-bd01-91c3ee701a4b","Type":"ContainerDied","Data":"54798133e3704ec612ff729a383c7033df8a5aece3e08b9f5b19c6282bfbb61f"} Oct 30 21:57:19.030605 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.030529 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54798133e3704ec612ff729a383c7033df8a5aece3e08b9f5b19c6282bfbb61f" Oct 30 21:57:19.191175 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.191142 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:57:19.206975 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.206942 4926 scope.go:122] "RemoveContainer" containerID="d3c7251a96bb42a51d8d58b3612f30c215b6d54e10043315d63ca5d429f8ec4e" Oct 30 21:57:19.220861 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.220836 4926 scope.go:122] "RemoveContainer" containerID="3355b38126e4007fe95ac0368957e0a2071848a3e81514c7965bcb13bf1dd9a2" Oct 30 21:57:19.245650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.245173 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-data\") pod \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " Oct 30 21:57:19.245650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.245243 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-credential-socket\") pod \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " Oct 30 21:57:19.245650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.245288 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-envoy\") pod \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " Oct 30 21:57:19.245650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.245348 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-workload-certs\") pod \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " Oct 30 21:57:19.245650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.245436 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-kube-api-access-mdpqt\" (UniqueName: \"kubernetes.io/projected/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-kube-api-access-mdpqt\") pod \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " Oct 30 21:57:19.245650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.245485 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-token\" (UniqueName: \"kubernetes.io/projected/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-token\") pod \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " Oct 30 21:57:19.245650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.245577 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istiod-ca-cert\") pod \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " Oct 30 21:57:19.245650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.245607 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-workload-socket\") pod \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " Oct 30 21:57:19.246847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.245683 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-podinfo\") pod \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\" (UID: \"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6\") " Oct 30 21:57:19.246847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.246131 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-credential-socket" pod "e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" (UID: "e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:19.246847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.246389 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:19.247289 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.247100 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-data" pod "e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" (UID: "e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:19.247458 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.247429 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-workload-certs" pod "e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" (UID: "e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:19.247876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.247624 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-workload-socket" pod "e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" (UID: "e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:19.247876 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.247785 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istiod-ca-cert" pod "e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" (UID: "e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:19.249730 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.249696 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-envoy" pod "e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" (UID: "e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:19.251008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.250985 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-token" pod "e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" (UID: "e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:19.251528 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.251506 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-kube-api-access-mdpqt" pod "e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" (UID: "e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6"). InnerVolumeSpecName "kube-api-access-mdpqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:19.252692 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.252658 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-podinfo" pod "e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" (UID: "e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:19.256176 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.256150 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54e7e6b-4l64w" Oct 30 21:57:19.268547 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.268512 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg" Oct 30 21:57:19.346762 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.346732 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cae23f38-df9f-443a-bf6b-aa20e5e365cd-kube-api-access-spdsh\" (UniqueName: \"kubernetes.io/projected/cae23f38-df9f-443a-bf6b-aa20e5e365cd-kube-api-access-spdsh\") pod \"cae23f38-df9f-443a-bf6b-aa20e5e365cd\" (UID: \"cae23f38-df9f-443a-bf6b-aa20e5e365cd\") " Oct 30 21:57:19.346930 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.346781 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e89d855c-7de9-409a-9644-474afb490832-kube-api-access-2c268\" (UniqueName: \"kubernetes.io/projected/e89d855c-7de9-409a-9644-474afb490832-kube-api-access-2c268\") pod \"e89d855c-7de9-409a-9644-474afb490832\" (UID: \"e89d855c-7de9-409a-9644-474afb490832\") " Oct 30 21:57:19.346930 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.346904 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e89d855c-7de9-409a-9644-474afb490832-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89d855c-7de9-409a-9644-474afb490832-maas-api-xks-ca\") pod \"e89d855c-7de9-409a-9644-474afb490832\" (UID: \"e89d855c-7de9-409a-9644-474afb490832\") " Oct 30 21:57:19.347081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.347020 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cae23f38-df9f-443a-bf6b-aa20e5e365cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cae23f38-df9f-443a-bf6b-aa20e5e365cd-maas-api-xks-ca\") pod \"cae23f38-df9f-443a-bf6b-aa20e5e365cd\" (UID: \"cae23f38-df9f-443a-bf6b-aa20e5e365cd\") " Oct 30 21:57:19.347081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.347073 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e89d855c-7de9-409a-9644-474afb490832-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89d855c-7de9-409a-9644-474afb490832-maas-api-openshift-ca\") pod \"e89d855c-7de9-409a-9644-474afb490832\" (UID: \"e89d855c-7de9-409a-9644-474afb490832\") " Oct 30 21:57:19.347212 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.347105 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cae23f38-df9f-443a-bf6b-aa20e5e365cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cae23f38-df9f-443a-bf6b-aa20e5e365cd-maas-api-openshift-ca\") pod \"cae23f38-df9f-443a-bf6b-aa20e5e365cd\" (UID: \"cae23f38-df9f-443a-bf6b-aa20e5e365cd\") " Oct 30 21:57:19.347561 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.347528 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e89d855c-7de9-409a-9644-474afb490832-maas-api-openshift-ca" pod "e89d855c-7de9-409a-9644-474afb490832" (UID: "e89d855c-7de9-409a-9644-474afb490832"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:19.347675 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.347615 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:19.347675 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.347641 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:19.347675 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.347655 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:19.347675 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.347668 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:19.347901 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.347680 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:19.347901 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.347693 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:19.347901 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.347707 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:19.347901 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.347721 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdpqt\" (UniqueName: \"kubernetes.io/projected/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6-kube-api-access-mdpqt\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:19.347901 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.347724 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cae23f38-df9f-443a-bf6b-aa20e5e365cd-maas-api-openshift-ca" pod "cae23f38-df9f-443a-bf6b-aa20e5e365cd" (UID: "cae23f38-df9f-443a-bf6b-aa20e5e365cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:19.349496 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.349469 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e89d855c-7de9-409a-9644-474afb490832-maas-api-xks-ca" pod "e89d855c-7de9-409a-9644-474afb490832" (UID: "e89d855c-7de9-409a-9644-474afb490832"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:19.349584 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.349523 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cae23f38-df9f-443a-bf6b-aa20e5e365cd-maas-api-xks-ca" pod "cae23f38-df9f-443a-bf6b-aa20e5e365cd" (UID: "cae23f38-df9f-443a-bf6b-aa20e5e365cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:19.349656 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.349632 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae23f38-df9f-443a-bf6b-aa20e5e365cd-kube-api-access-spdsh" pod "cae23f38-df9f-443a-bf6b-aa20e5e365cd" (UID: "cae23f38-df9f-443a-bf6b-aa20e5e365cd"). InnerVolumeSpecName "kube-api-access-spdsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:19.350106 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.350084 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e89d855c-7de9-409a-9644-474afb490832-kube-api-access-2c268" pod "e89d855c-7de9-409a-9644-474afb490832" (UID: "e89d855c-7de9-409a-9644-474afb490832"). InnerVolumeSpecName "kube-api-access-2c268". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:19.448577 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.448541 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cae23f38-df9f-443a-bf6b-aa20e5e365cd-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:19.448577 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.448572 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e89d855c-7de9-409a-9644-474afb490832-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:19.448577 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.448583 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cae23f38-df9f-443a-bf6b-aa20e5e365cd-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:19.448840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.448593 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spdsh\" (UniqueName: \"kubernetes.io/projected/cae23f38-df9f-443a-bf6b-aa20e5e365cd-kube-api-access-spdsh\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:19.448840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.448603 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c268\" (UniqueName: \"kubernetes.io/projected/e89d855c-7de9-409a-9644-474afb490832-kube-api-access-2c268\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:19.448840 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:19.448614 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e89d855c-7de9-409a-9644-474afb490832-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:20.041706 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:20.041668 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" Oct 30 21:57:20.041925 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:20.041709 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45" event={"ID":"e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6","Type":"ContainerDied","Data":"3d3368b091b41345e306ac328c1e459b79a91d0122903af16cd9840f813962bf"} Oct 30 21:57:20.041925 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:20.041769 4926 scope.go:122] "RemoveContainer" containerID="6556f37eef023f1c2c84e045cab721271550a5488205f608f502a2bb40dd1574" Oct 30 21:57:20.043417 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:20.043385 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54e7e6b-4l64w" Oct 30 21:57:20.043534 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:20.043442 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54e7e6b-4l64w" event={"ID":"cae23f38-df9f-443a-bf6b-aa20e5e365cd","Type":"ContainerDied","Data":"a3006ef0c462269206296f0284b44e63556444f863a6222fc6626a000abb58df"} Oct 30 21:57:20.043534 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:20.043497 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3006ef0c462269206296f0284b44e63556444f863a6222fc6626a000abb58df" Oct 30 21:57:20.045478 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:20.045369 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg" Oct 30 21:57:20.045478 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:20.045401 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg" event={"ID":"e89d855c-7de9-409a-9644-474afb490832","Type":"ContainerDied","Data":"b624f33e33efa5f611b92ee488e47184506f2279d1113a7fc575a381472ef24e"} Oct 30 21:57:20.045478 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:20.045446 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b624f33e33efa5f611b92ee488e47184506f2279d1113a7fc575a381472ef24e" Oct 30 21:57:20.099083 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:20.099046 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45"] Oct 30 21:57:20.110806 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:20.110766 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-be3ada-openshift-default-6878f56887-xnt45"] Oct 30 21:57:21.016615 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:21.016582 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" path="/var/lib/kubelet/pods/e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6/volumes" Oct 30 21:57:21.538866 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:21.538818 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" containerID="cri-o://b7fb8a889ab78ee1dbe8968ffd176fe7a6866e51d2b1bd1121af6c180209e9a2" gracePeriod=45 Oct 30 21:57:21.827334 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:21.827303 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:57:21.876779 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:21.876737 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-tmp-dir\") pod \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " Oct 30 21:57:21.876960 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:21.876894 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-kube-api-access-8twjb\" (UniqueName: \"kubernetes.io/projected/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-kube-api-access-8twjb\") pod \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " Oct 30 21:57:21.876960 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:21.876939 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-tls-certs\") pod \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " Oct 30 21:57:21.877143 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:21.876985 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-model-cache\") pod \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " Oct 30 21:57:21.877143 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:21.877104 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-home\" (UniqueName: \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-home\") pod \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " Oct 30 21:57:21.877252 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:21.877156 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-dshm\") pod \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\" (UID: \"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4\") " Oct 30 21:57:21.878658 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:21.878446 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-tmp-dir" pod "bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" (UID: "bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:21.878791 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:21.878654 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-model-cache" pod "bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" (UID: "bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:21.879216 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:21.879188 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-home" pod "bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" (UID: "bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:21.883001 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:21.882821 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-kube-api-access-8twjb" pod "bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" (UID: "bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4"). InnerVolumeSpecName "kube-api-access-8twjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:21.904544 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:21.904487 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-dshm" pod "bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" (UID: "bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:21.915608 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:21.915544 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-tls-certs" pod "bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" (UID: "bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:21.978682 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:21.978635 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8twjb\" (UniqueName: \"kubernetes.io/projected/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-kube-api-access-8twjb\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:21.978682 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:21.978680 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:21.978908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:21.978697 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:21.978908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:21.978711 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:21.978908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:21.978724 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:21.978908 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:21.978736 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:22.064947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:22.064849 4926 generic.go:356] "Generic (PLEG): container finished" podID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerID="b7fb8a889ab78ee1dbe8968ffd176fe7a6866e51d2b1bd1121af6c180209e9a2" exitCode=0 Oct 30 21:57:22.064947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:22.064908 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" Oct 30 21:57:22.064947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:22.064935 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" event={"ID":"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4","Type":"ContainerDied","Data":"b7fb8a889ab78ee1dbe8968ffd176fe7a6866e51d2b1bd1121af6c180209e9a2"} Oct 30 21:57:22.065544 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:22.064986 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm" event={"ID":"bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4","Type":"ContainerDied","Data":"6c6d16153b84bfb98e35d37a8f08402c6893df248469d2a07b20a583d296f9fc"} Oct 30 21:57:22.065544 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:22.065011 4926 scope.go:122] "RemoveContainer" containerID="b7fb8a889ab78ee1dbe8968ffd176fe7a6866e51d2b1bd1121af6c180209e9a2" Oct 30 21:57:22.077250 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:22.077225 4926 scope.go:122] "RemoveContainer" containerID="b7fb8a889ab78ee1dbe8968ffd176fe7a6866e51d2b1bd1121af6c180209e9a2" Oct 30 21:57:22.077729 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:57:22.077576 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7fb8a889ab78ee1dbe8968ffd176fe7a6866e51d2b1bd1121af6c180209e9a2\": container with ID starting with b7fb8a889ab78ee1dbe8968ffd176fe7a6866e51d2b1bd1121af6c180209e9a2 not found: ID does not exist" containerID="b7fb8a889ab78ee1dbe8968ffd176fe7a6866e51d2b1bd1121af6c180209e9a2" Oct 30 21:57:22.077729 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:22.077618 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7fb8a889ab78ee1dbe8968ffd176fe7a6866e51d2b1bd1121af6c180209e9a2"} err="failed to get container status \"b7fb8a889ab78ee1dbe8968ffd176fe7a6866e51d2b1bd1121af6c180209e9a2\": rpc error: code = NotFound desc = could not find container \"b7fb8a889ab78ee1dbe8968ffd176fe7a6866e51d2b1bd1121af6c180209e9a2\": container with ID starting with b7fb8a889ab78ee1dbe8968ffd176fe7a6866e51d2b1bd1121af6c180209e9a2 not found: ID does not exist" Oct 30 21:57:22.114150 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:22.114110 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm"] Oct 30 21:57:22.117920 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:22.117893 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-839163/simulator-w4-839163-kserve-5c66cb4d9f-sf5fm"] Oct 30 21:57:23.018177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:23.018142 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" path="/var/lib/kubelet/pods/bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4/volumes" Oct 30 21:57:23.289173 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:23.289065 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft" podUID="11be824c-778f-4301-b2c5-7b0c8ca3d04c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:57:26.755639 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:26.755589 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w"] Oct 30 21:57:26.761795 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:26.761766 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-da52fd-bh44w"] Oct 30 21:57:27.015025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:27.014946 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7d8d1ce-693f-4751-b6fe-ad4774673512" path="/var/lib/kubelet/pods/a7d8d1ce-693f-4751-b6fe-ad4774673512/volumes" Oct 30 21:57:27.230602 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:27.230565 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7"] Oct 30 21:57:27.238688 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:27.238657 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-839163-9njp7"] Oct 30 21:57:29.015494 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:29.015456 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18" path="/var/lib/kubelet/pods/d9ea34a7-cc8b-4e21-9aab-ac38fa8e2d18/volumes" Oct 30 21:57:33.289941 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:33.289872 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft" podUID="11be824c-778f-4301-b2c5-7b0c8ca3d04c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 21:57:33.654403 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:33.654312 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:57:33.986970 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:33.986929 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7"] Oct 30 21:57:34.018057 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:34.007915 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-480d41-c5pq7"] Oct 30 21:57:35.015919 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:35.015883 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25313085-efd3-4732-bd01-91c3ee701a4b" path="/var/lib/kubelet/pods/25313085-efd3-4732-bd01-91c3ee701a4b/volumes" Oct 30 21:57:36.621834 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:36.621794 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf"] Oct 30 21:57:36.622437 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:36.622003 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" podUID="f40cfadc-67d6-467e-bced-347b09dd6462" containerName="istio-proxy" containerID="cri-o://b5c917de81666e2c397c8dbe044a742efa7b64a2643d69d6548e452235e8fb0b" gracePeriod=30 Oct 30 21:57:37.143219 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.143183 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44"] Oct 30 21:57:37.143636 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.143622 4926 cpu_manager.go:414] "containerMap: removing container" podUID="25313085-efd3-4732-bd01-91c3ee701a4b" containerName="revoke-keys" Oct 30 21:57:37.143685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.143637 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25313085-efd3-4732-bd01-91c3ee701a4b" containerName="revoke-keys" Oct 30 21:57:37.143685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.143663 4926 cpu_manager.go:414] "containerMap: removing container" podUID="e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" containerName="istio-proxy" Oct 30 21:57:37.143685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.143669 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" containerName="istio-proxy" Oct 30 21:57:37.143790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.143696 4926 cpu_manager.go:414] "containerMap: removing container" podUID="586f88e9-d006-46ea-bef3-68cc117aaa81" containerName="payload-processing" Oct 30 21:57:37.143790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.143702 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="586f88e9-d006-46ea-bef3-68cc117aaa81" containerName="payload-processing" Oct 30 21:57:37.143790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.143709 4926 cpu_manager.go:414] "containerMap: removing container" podUID="e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" containerName="istio-proxy" Oct 30 21:57:37.143790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.143714 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" containerName="istio-proxy" Oct 30 21:57:37.143790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.143725 4926 cpu_manager.go:414] "containerMap: removing container" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" Oct 30 21:57:37.143790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.143730 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" Oct 30 21:57:37.143790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.143741 4926 cpu_manager.go:414] "containerMap: removing container" podUID="cae23f38-df9f-443a-bf6b-aa20e5e365cd" containerName="revoke-keys" Oct 30 21:57:37.143790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.143745 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cae23f38-df9f-443a-bf6b-aa20e5e365cd" containerName="revoke-keys" Oct 30 21:57:37.143790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.143762 4926 cpu_manager.go:414] "containerMap: removing container" podUID="e89d855c-7de9-409a-9644-474afb490832" containerName="revoke-keys" Oct 30 21:57:37.143790 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.143767 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e89d855c-7de9-409a-9644-474afb490832" containerName="revoke-keys" Oct 30 21:57:37.144132 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.143835 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" containerName="istio-proxy" Oct 30 21:57:37.144132 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.143846 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="586f88e9-d006-46ea-bef3-68cc117aaa81" containerName="payload-processing" Oct 30 21:57:37.144132 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.143853 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7ff8991-1ebf-4cf8-96c5-bbc0fd96d6a6" containerName="istio-proxy" Oct 30 21:57:37.144132 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.143859 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbb5e6e6-a437-491d-9d1c-a1bfe6b258f4" containerName="main" Oct 30 21:57:37.144132 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.143867 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="cae23f38-df9f-443a-bf6b-aa20e5e365cd" containerName="revoke-keys" Oct 30 21:57:37.144132 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.143874 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="e89d855c-7de9-409a-9644-474afb490832" containerName="revoke-keys" Oct 30 21:57:37.144132 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.143881 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="25313085-efd3-4732-bd01-91c3ee701a4b" containerName="revoke-keys" Oct 30 21:57:37.148872 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.148852 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44" Oct 30 21:57:37.164814 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.164785 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:57:37.165294 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.165249 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:57:37.168472 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.168446 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44"] Oct 30 21:57:37.168894 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.168873 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:57:37.173370 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.173349 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:57:37.256650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.256608 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ba0c21-248b-4f6b-889a-78ac94c2a07e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44\" (UID: \"06ba0c21-248b-4f6b-889a-78ac94c2a07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44" Oct 30 21:57:37.256816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.256665 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glnhb\" (UniqueName: \"kubernetes.io/projected/06ba0c21-248b-4f6b-889a-78ac94c2a07e-kube-api-access-glnhb\") pod \"maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44\" (UID: \"06ba0c21-248b-4f6b-889a-78ac94c2a07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44" Oct 30 21:57:37.256816 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.256795 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ba0c21-248b-4f6b-889a-78ac94c2a07e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44\" (UID: \"06ba0c21-248b-4f6b-889a-78ac94c2a07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44" Oct 30 21:57:37.358444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.358408 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ba0c21-248b-4f6b-889a-78ac94c2a07e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44\" (UID: \"06ba0c21-248b-4f6b-889a-78ac94c2a07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44" Oct 30 21:57:37.358639 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.358462 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glnhb\" (UniqueName: \"kubernetes.io/projected/06ba0c21-248b-4f6b-889a-78ac94c2a07e-kube-api-access-glnhb\") pod \"maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44\" (UID: \"06ba0c21-248b-4f6b-889a-78ac94c2a07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44" Oct 30 21:57:37.358639 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.358548 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ba0c21-248b-4f6b-889a-78ac94c2a07e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44\" (UID: \"06ba0c21-248b-4f6b-889a-78ac94c2a07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44" Oct 30 21:57:37.359286 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.359259 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ba0c21-248b-4f6b-889a-78ac94c2a07e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44\" (UID: \"06ba0c21-248b-4f6b-889a-78ac94c2a07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44" Oct 30 21:57:37.361848 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.361783 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ba0c21-248b-4f6b-889a-78ac94c2a07e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44\" (UID: \"06ba0c21-248b-4f6b-889a-78ac94c2a07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44" Oct 30 21:57:37.400104 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.399973 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glnhb\" (UniqueName: \"kubernetes.io/projected/06ba0c21-248b-4f6b-889a-78ac94c2a07e-kube-api-access-glnhb\") pod \"maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44\" (UID: \"06ba0c21-248b-4f6b-889a-78ac94c2a07e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44" Oct 30 21:57:37.463545 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.463242 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44" Oct 30 21:57:37.554854 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:57:37.554815 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:57:37.567829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.567800 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft" Oct 30 21:57:37.664347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.663993 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11be824c-778f-4301-b2c5-7b0c8ca3d04c-kube-api-access-2rwwf\" (UniqueName: \"kubernetes.io/projected/11be824c-778f-4301-b2c5-7b0c8ca3d04c-kube-api-access-2rwwf\") pod \"11be824c-778f-4301-b2c5-7b0c8ca3d04c\" (UID: \"11be824c-778f-4301-b2c5-7b0c8ca3d04c\") " Oct 30 21:57:37.664347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.664089 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11be824c-778f-4301-b2c5-7b0c8ca3d04c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11be824c-778f-4301-b2c5-7b0c8ca3d04c-plugins-config-volume\") pod \"11be824c-778f-4301-b2c5-7b0c8ca3d04c\" (UID: \"11be824c-778f-4301-b2c5-7b0c8ca3d04c\") " Oct 30 21:57:37.664785 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.664590 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11be824c-778f-4301-b2c5-7b0c8ca3d04c-plugins-config-volume" pod "11be824c-778f-4301-b2c5-7b0c8ca3d04c" (UID: "11be824c-778f-4301-b2c5-7b0c8ca3d04c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:37.666713 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.666682 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11be824c-778f-4301-b2c5-7b0c8ca3d04c-kube-api-access-2rwwf" pod "11be824c-778f-4301-b2c5-7b0c8ca3d04c" (UID: "11be824c-778f-4301-b2c5-7b0c8ca3d04c"). InnerVolumeSpecName "kube-api-access-2rwwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:37.765267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.765229 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rwwf\" (UniqueName: \"kubernetes.io/projected/11be824c-778f-4301-b2c5-7b0c8ca3d04c-kube-api-access-2rwwf\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:37.765267 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.765261 4926 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11be824c-778f-4301-b2c5-7b0c8ca3d04c-plugins-config-volume\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:37.864273 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:37.864240 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44"] Oct 30 21:57:37.865696 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:57:37.865666 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ba0c21_248b_4f6b_889a_78ac94c2a07e.slice/crio-3fe77b58d06f42c6bce829d8c0bde83d9b001724c110922ab6ea9885218f835b WatchSource:0}: Error finding container 3fe77b58d06f42c6bce829d8c0bde83d9b001724c110922ab6ea9885218f835b: Status 404 returned error can't find the container with id 3fe77b58d06f42c6bce829d8c0bde83d9b001724c110922ab6ea9885218f835b Oct 30 21:57:38.198433 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:38.198396 4926 generic.go:356] "Generic (PLEG): container finished" podID="11be824c-778f-4301-b2c5-7b0c8ca3d04c" containerID="42e68838bfebe24dff828eb4163541fa2404849a78b58a906e0f21ac908e3df0" exitCode=0 Oct 30 21:57:38.198612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:38.198436 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft" Oct 30 21:57:38.198612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:38.198472 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft" event={"ID":"11be824c-778f-4301-b2c5-7b0c8ca3d04c","Type":"ContainerDied","Data":"42e68838bfebe24dff828eb4163541fa2404849a78b58a906e0f21ac908e3df0"} Oct 30 21:57:38.198612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:38.198524 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft" event={"ID":"11be824c-778f-4301-b2c5-7b0c8ca3d04c","Type":"ContainerDied","Data":"dbc6c5ac60af8d82e0e7caedbe8610124ec7a1bef7420643df3fc4ccaf863967"} Oct 30 21:57:38.198612 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:38.198556 4926 scope.go:122] "RemoveContainer" containerID="42e68838bfebe24dff828eb4163541fa2404849a78b58a906e0f21ac908e3df0" Oct 30 21:57:38.200694 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:38.200658 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44" event={"ID":"06ba0c21-248b-4f6b-889a-78ac94c2a07e","Type":"ContainerStarted","Data":"af6e2f94f5cb14aa266b017de366b14778b0c011662810f28c353fc537f0cd41"} Oct 30 21:57:38.200829 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:38.200735 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44" event={"ID":"06ba0c21-248b-4f6b-889a-78ac94c2a07e","Type":"ContainerStarted","Data":"3fe77b58d06f42c6bce829d8c0bde83d9b001724c110922ab6ea9885218f835b"} Oct 30 21:57:38.210222 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:38.210198 4926 scope.go:122] "RemoveContainer" containerID="42e68838bfebe24dff828eb4163541fa2404849a78b58a906e0f21ac908e3df0" Oct 30 21:57:38.210513 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:57:38.210492 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42e68838bfebe24dff828eb4163541fa2404849a78b58a906e0f21ac908e3df0\": container with ID starting with 42e68838bfebe24dff828eb4163541fa2404849a78b58a906e0f21ac908e3df0 not found: ID does not exist" containerID="42e68838bfebe24dff828eb4163541fa2404849a78b58a906e0f21ac908e3df0" Oct 30 21:57:38.210578 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:38.210525 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42e68838bfebe24dff828eb4163541fa2404849a78b58a906e0f21ac908e3df0"} err="failed to get container status \"42e68838bfebe24dff828eb4163541fa2404849a78b58a906e0f21ac908e3df0\": rpc error: code = NotFound desc = could not find container \"42e68838bfebe24dff828eb4163541fa2404849a78b58a906e0f21ac908e3df0\": container with ID starting with 42e68838bfebe24dff828eb4163541fa2404849a78b58a906e0f21ac908e3df0 not found: ID does not exist" Oct 30 21:57:38.251856 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:38.251821 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft"] Oct 30 21:57:38.261590 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:38.261557 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-839163-6bb9bf88c-hkdft"] Oct 30 21:57:38.307492 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:38.307430 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44" podStartSLOduration=1.307416779 podStartE2EDuration="1.307416779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:38.303380738 +0000 UTC m=+3445.424454485" watchObservedRunningTime="2026-10-30 21:57:38.307416779 +0000 UTC m=+3445.428490525" Oct 30 21:57:39.015765 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:39.015730 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11be824c-778f-4301-b2c5-7b0c8ca3d04c" path="/var/lib/kubelet/pods/11be824c-778f-4301-b2c5-7b0c8ca3d04c/volumes" Oct 30 21:57:39.212316 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:39.212281 4926 generic.go:356] "Generic (PLEG): container finished" podID="06ba0c21-248b-4f6b-889a-78ac94c2a07e" containerID="af6e2f94f5cb14aa266b017de366b14778b0c011662810f28c353fc537f0cd41" exitCode=0 Oct 30 21:57:39.212510 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:39.212366 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44" event={"ID":"06ba0c21-248b-4f6b-889a-78ac94c2a07e","Type":"ContainerDied","Data":"af6e2f94f5cb14aa266b017de366b14778b0c011662810f28c353fc537f0cd41"} Oct 30 21:57:40.379450 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:40.379423 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44" Oct 30 21:57:40.494324 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:40.494237 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ba0c21-248b-4f6b-889a-78ac94c2a07e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ba0c21-248b-4f6b-889a-78ac94c2a07e-maas-api-openshift-ca\") pod \"06ba0c21-248b-4f6b-889a-78ac94c2a07e\" (UID: \"06ba0c21-248b-4f6b-889a-78ac94c2a07e\") " Oct 30 21:57:40.494498 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:40.494432 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ba0c21-248b-4f6b-889a-78ac94c2a07e-kube-api-access-glnhb\" (UniqueName: \"kubernetes.io/projected/06ba0c21-248b-4f6b-889a-78ac94c2a07e-kube-api-access-glnhb\") pod \"06ba0c21-248b-4f6b-889a-78ac94c2a07e\" (UID: \"06ba0c21-248b-4f6b-889a-78ac94c2a07e\") " Oct 30 21:57:40.494547 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:40.494523 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06ba0c21-248b-4f6b-889a-78ac94c2a07e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ba0c21-248b-4f6b-889a-78ac94c2a07e-maas-api-xks-ca\") pod \"06ba0c21-248b-4f6b-889a-78ac94c2a07e\" (UID: \"06ba0c21-248b-4f6b-889a-78ac94c2a07e\") " Oct 30 21:57:40.494657 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:40.494629 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ba0c21-248b-4f6b-889a-78ac94c2a07e-maas-api-openshift-ca" pod "06ba0c21-248b-4f6b-889a-78ac94c2a07e" (UID: "06ba0c21-248b-4f6b-889a-78ac94c2a07e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:40.494955 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:40.494928 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ba0c21-248b-4f6b-889a-78ac94c2a07e-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:40.496755 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:40.496720 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ba0c21-248b-4f6b-889a-78ac94c2a07e-maas-api-xks-ca" pod "06ba0c21-248b-4f6b-889a-78ac94c2a07e" (UID: "06ba0c21-248b-4f6b-889a-78ac94c2a07e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:40.497065 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:40.497023 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ba0c21-248b-4f6b-889a-78ac94c2a07e-kube-api-access-glnhb" pod "06ba0c21-248b-4f6b-889a-78ac94c2a07e" (UID: "06ba0c21-248b-4f6b-889a-78ac94c2a07e"). InnerVolumeSpecName "kube-api-access-glnhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:40.596484 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:40.596445 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ba0c21-248b-4f6b-889a-78ac94c2a07e-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:40.596484 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:40.596479 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glnhb\" (UniqueName: \"kubernetes.io/projected/06ba0c21-248b-4f6b-889a-78ac94c2a07e-kube-api-access-glnhb\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:41.231827 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:41.231792 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44" Oct 30 21:57:41.232007 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:41.231791 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44" event={"ID":"06ba0c21-248b-4f6b-889a-78ac94c2a07e","Type":"ContainerDied","Data":"3fe77b58d06f42c6bce829d8c0bde83d9b001724c110922ab6ea9885218f835b"} Oct 30 21:57:41.232007 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:41.231929 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fe77b58d06f42c6bce829d8c0bde83d9b001724c110922ab6ea9885218f835b" Oct 30 21:57:41.864745 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:41.864707 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps"] Oct 30 21:57:41.878076 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:41.878016 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr"] Oct 30 21:57:41.886688 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:41.886662 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:57:41.901672 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:41.901596 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx"] Oct 30 21:57:42.015961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.013145 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-credential-socket\") pod \"f40cfadc-67d6-467e-bced-347b09dd6462\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " Oct 30 21:57:42.015961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.013216 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40cfadc-67d6-467e-bced-347b09dd6462-istio-token\" (UniqueName: \"kubernetes.io/projected/f40cfadc-67d6-467e-bced-347b09dd6462-istio-token\") pod \"f40cfadc-67d6-467e-bced-347b09dd6462\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " Oct 30 21:57:42.015961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.013252 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-istio-envoy\") pod \"f40cfadc-67d6-467e-bced-347b09dd6462\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " Oct 30 21:57:42.015961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.013283 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f40cfadc-67d6-467e-bced-347b09dd6462-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40cfadc-67d6-467e-bced-347b09dd6462-istio-podinfo\") pod \"f40cfadc-67d6-467e-bced-347b09dd6462\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " Oct 30 21:57:42.015961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.013330 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f40cfadc-67d6-467e-bced-347b09dd6462-kube-api-access-mq6xm\" (UniqueName: \"kubernetes.io/projected/f40cfadc-67d6-467e-bced-347b09dd6462-kube-api-access-mq6xm\") pod \"f40cfadc-67d6-467e-bced-347b09dd6462\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " Oct 30 21:57:42.015961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.013434 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-workload-socket\") pod \"f40cfadc-67d6-467e-bced-347b09dd6462\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " Oct 30 21:57:42.015961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.013485 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-workload-certs\") pod \"f40cfadc-67d6-467e-bced-347b09dd6462\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " Oct 30 21:57:42.015961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.013569 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-istio-data\") pod \"f40cfadc-67d6-467e-bced-347b09dd6462\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " Oct 30 21:57:42.015961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.013597 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f40cfadc-67d6-467e-bced-347b09dd6462-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40cfadc-67d6-467e-bced-347b09dd6462-istiod-ca-cert\") pod \"f40cfadc-67d6-467e-bced-347b09dd6462\" (UID: \"f40cfadc-67d6-467e-bced-347b09dd6462\") " Oct 30 21:57:42.015961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.014440 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-credential-socket" pod "f40cfadc-67d6-467e-bced-347b09dd6462" (UID: "f40cfadc-67d6-467e-bced-347b09dd6462"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:42.015961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.014514 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f40cfadc-67d6-467e-bced-347b09dd6462-istiod-ca-cert" pod "f40cfadc-67d6-467e-bced-347b09dd6462" (UID: "f40cfadc-67d6-467e-bced-347b09dd6462"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:42.015961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.015196 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-workload-certs" pod "f40cfadc-67d6-467e-bced-347b09dd6462" (UID: "f40cfadc-67d6-467e-bced-347b09dd6462"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:42.015961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.015360 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-istio-data" pod "f40cfadc-67d6-467e-bced-347b09dd6462" (UID: "f40cfadc-67d6-467e-bced-347b09dd6462"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:42.015961 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.015382 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-workload-socket" pod "f40cfadc-67d6-467e-bced-347b09dd6462" (UID: "f40cfadc-67d6-467e-bced-347b09dd6462"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:42.018396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.017236 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40cfadc-67d6-467e-bced-347b09dd6462-istio-token" pod "f40cfadc-67d6-467e-bced-347b09dd6462" (UID: "f40cfadc-67d6-467e-bced-347b09dd6462"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:42.018396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.017748 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f40cfadc-67d6-467e-bced-347b09dd6462-kube-api-access-mq6xm" pod "f40cfadc-67d6-467e-bced-347b09dd6462" (UID: "f40cfadc-67d6-467e-bced-347b09dd6462"). InnerVolumeSpecName "kube-api-access-mq6xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:42.046398 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.038482 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f40cfadc-67d6-467e-bced-347b09dd6462-istio-podinfo" pod "f40cfadc-67d6-467e-bced-347b09dd6462" (UID: "f40cfadc-67d6-467e-bced-347b09dd6462"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:57:42.046398 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.042468 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-istio-envoy" pod "f40cfadc-67d6-467e-bced-347b09dd6462" (UID: "f40cfadc-67d6-467e-bced-347b09dd6462"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:42.048936 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.048899 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5"] Oct 30 21:57:42.049540 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.049521 4926 cpu_manager.go:414] "containerMap: removing container" podUID="f40cfadc-67d6-467e-bced-347b09dd6462" containerName="istio-proxy" Oct 30 21:57:42.049540 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.049540 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f40cfadc-67d6-467e-bced-347b09dd6462" containerName="istio-proxy" Oct 30 21:57:42.049726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.049555 4926 cpu_manager.go:414] "containerMap: removing container" podUID="06ba0c21-248b-4f6b-889a-78ac94c2a07e" containerName="revoke-keys" Oct 30 21:57:42.049726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.049561 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ba0c21-248b-4f6b-889a-78ac94c2a07e" containerName="revoke-keys" Oct 30 21:57:42.049726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.049580 4926 cpu_manager.go:414] "containerMap: removing container" podUID="f40cfadc-67d6-467e-bced-347b09dd6462" containerName="istio-proxy" Oct 30 21:57:42.049726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.049585 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f40cfadc-67d6-467e-bced-347b09dd6462" containerName="istio-proxy" Oct 30 21:57:42.049726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.049592 4926 cpu_manager.go:414] "containerMap: removing container" podUID="11be824c-778f-4301-b2c5-7b0c8ca3d04c" containerName="payload-pre-processing" Oct 30 21:57:42.049726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.049598 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11be824c-778f-4301-b2c5-7b0c8ca3d04c" containerName="payload-pre-processing" Oct 30 21:57:42.049726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.049664 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="f40cfadc-67d6-467e-bced-347b09dd6462" containerName="istio-proxy" Oct 30 21:57:42.049726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.049673 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ba0c21-248b-4f6b-889a-78ac94c2a07e" containerName="revoke-keys" Oct 30 21:57:42.049726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.049684 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="11be824c-778f-4301-b2c5-7b0c8ca3d04c" containerName="payload-pre-processing" Oct 30 21:57:42.053210 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.053188 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5" Oct 30 21:57:42.074290 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.074265 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:57:42.076947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.076927 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:57:42.083594 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.083566 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5"] Oct 30 21:57:42.095475 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.095452 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:57:42.095666 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.095652 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:57:42.114550 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.114517 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:42.114550 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.114546 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f40cfadc-67d6-467e-bced-347b09dd6462-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:42.114757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.114557 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:42.114757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.114566 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f40cfadc-67d6-467e-bced-347b09dd6462-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:42.114757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.114576 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:42.114757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.114584 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f40cfadc-67d6-467e-bced-347b09dd6462-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:42.114757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.114592 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq6xm\" (UniqueName: \"kubernetes.io/projected/f40cfadc-67d6-467e-bced-347b09dd6462-kube-api-access-mq6xm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:42.114757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.114603 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:42.114757 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.114611 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f40cfadc-67d6-467e-bced-347b09dd6462-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:42.215356 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.215322 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5\" (UID: \"fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5" Oct 30 21:57:42.215563 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.215384 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqdbj\" (UniqueName: \"kubernetes.io/projected/fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6-kube-api-access-gqdbj\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5\" (UID: \"fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5" Oct 30 21:57:42.215563 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.215493 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5\" (UID: \"fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5" Oct 30 21:57:42.241698 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.241658 4926 generic.go:356] "Generic (PLEG): container finished" podID="f40cfadc-67d6-467e-bced-347b09dd6462" containerID="b5c917de81666e2c397c8dbe044a742efa7b64a2643d69d6548e452235e8fb0b" exitCode=0 Oct 30 21:57:42.241938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.241736 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" Oct 30 21:57:42.241938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.241767 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" event={"ID":"f40cfadc-67d6-467e-bced-347b09dd6462","Type":"ContainerDied","Data":"b5c917de81666e2c397c8dbe044a742efa7b64a2643d69d6548e452235e8fb0b"} Oct 30 21:57:42.241938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.241858 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf" event={"ID":"f40cfadc-67d6-467e-bced-347b09dd6462","Type":"ContainerDied","Data":"42257197effc39e67d0f04332cc18aecb7e8a56c59ebd6ce0314615ad189cd0e"} Oct 30 21:57:42.241938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.241901 4926 scope.go:122] "RemoveContainer" containerID="b5c917de81666e2c397c8dbe044a742efa7b64a2643d69d6548e452235e8fb0b" Oct 30 21:57:42.253727 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.253703 4926 scope.go:122] "RemoveContainer" containerID="22d8b63f2e3fe08d22a83b30d53fe671845ff89e68f9a4d6f0cc1ca58206960b" Oct 30 21:57:42.265959 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.265938 4926 scope.go:122] "RemoveContainer" containerID="b5c917de81666e2c397c8dbe044a742efa7b64a2643d69d6548e452235e8fb0b" Oct 30 21:57:42.266317 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:57:42.266291 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5c917de81666e2c397c8dbe044a742efa7b64a2643d69d6548e452235e8fb0b\": container with ID starting with b5c917de81666e2c397c8dbe044a742efa7b64a2643d69d6548e452235e8fb0b not found: ID does not exist" containerID="b5c917de81666e2c397c8dbe044a742efa7b64a2643d69d6548e452235e8fb0b" Oct 30 21:57:42.266434 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.266331 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5c917de81666e2c397c8dbe044a742efa7b64a2643d69d6548e452235e8fb0b"} err="failed to get container status \"b5c917de81666e2c397c8dbe044a742efa7b64a2643d69d6548e452235e8fb0b\": rpc error: code = NotFound desc = could not find container \"b5c917de81666e2c397c8dbe044a742efa7b64a2643d69d6548e452235e8fb0b\": container with ID starting with b5c917de81666e2c397c8dbe044a742efa7b64a2643d69d6548e452235e8fb0b not found: ID does not exist" Oct 30 21:57:42.266434 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.266355 4926 scope.go:122] "RemoveContainer" containerID="22d8b63f2e3fe08d22a83b30d53fe671845ff89e68f9a4d6f0cc1ca58206960b" Oct 30 21:57:42.266665 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:57:42.266645 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22d8b63f2e3fe08d22a83b30d53fe671845ff89e68f9a4d6f0cc1ca58206960b\": container with ID starting with 22d8b63f2e3fe08d22a83b30d53fe671845ff89e68f9a4d6f0cc1ca58206960b not found: ID does not exist" containerID="22d8b63f2e3fe08d22a83b30d53fe671845ff89e68f9a4d6f0cc1ca58206960b" Oct 30 21:57:42.266712 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.266673 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22d8b63f2e3fe08d22a83b30d53fe671845ff89e68f9a4d6f0cc1ca58206960b"} err="failed to get container status \"22d8b63f2e3fe08d22a83b30d53fe671845ff89e68f9a4d6f0cc1ca58206960b\": rpc error: code = NotFound desc = could not find container \"22d8b63f2e3fe08d22a83b30d53fe671845ff89e68f9a4d6f0cc1ca58206960b\": container with ID starting with 22d8b63f2e3fe08d22a83b30d53fe671845ff89e68f9a4d6f0cc1ca58206960b not found: ID does not exist" Oct 30 21:57:42.307399 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.307366 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf"] Oct 30 21:57:42.317345 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.317304 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5\" (UID: \"fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5" Oct 30 21:57:42.317518 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.317401 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqdbj\" (UniqueName: \"kubernetes.io/projected/fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6-kube-api-access-gqdbj\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5\" (UID: \"fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5" Oct 30 21:57:42.317518 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.317455 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5\" (UID: \"fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5" Oct 30 21:57:42.318089 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.318065 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5\" (UID: \"fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5" Oct 30 21:57:42.320599 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.320569 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5\" (UID: \"fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5" Oct 30 21:57:42.322182 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.322159 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-da52fd-openshift-default-6bbb549bd7-7bgsf"] Oct 30 21:57:42.340343 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.340318 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqdbj\" (UniqueName: \"kubernetes.io/projected/fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6-kube-api-access-gqdbj\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5\" (UID: \"fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5" Oct 30 21:57:42.367136 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.367105 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5" Oct 30 21:57:42.676346 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:42.676310 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5"] Oct 30 21:57:42.677570 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:57:42.677540 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc35cd74_d73d_4a73_b1bb_07f8e51fe4d6.slice/crio-85e23af8a9a7da720d28964306e6e400ba902fd0bc92904de4e7e02e50101531 WatchSource:0}: Error finding container 85e23af8a9a7da720d28964306e6e400ba902fd0bc92904de4e7e02e50101531: Status 404 returned error can't find the container with id 85e23af8a9a7da720d28964306e6e400ba902fd0bc92904de4e7e02e50101531 Oct 30 21:57:43.016010 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:43.015971 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f40cfadc-67d6-467e-bced-347b09dd6462" path="/var/lib/kubelet/pods/f40cfadc-67d6-467e-bced-347b09dd6462/volumes" Oct 30 21:57:43.251467 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:43.251368 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5" event={"ID":"fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6","Type":"ContainerStarted","Data":"1ed8738b729ed9cb467df3980329faf4ea7c9d91419c3df816a92260bcb2b615"} Oct 30 21:57:43.251467 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:43.251432 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5" event={"ID":"fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6","Type":"ContainerStarted","Data":"85e23af8a9a7da720d28964306e6e400ba902fd0bc92904de4e7e02e50101531"} Oct 30 21:57:43.295046 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:43.294975 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5" podStartSLOduration=1.2949596890000001 podStartE2EDuration="1.294959689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:43.29201556 +0000 UTC m=+3450.413089310" watchObservedRunningTime="2026-10-30 21:57:43.294959689 +0000 UTC m=+3450.416033437" Oct 30 21:57:44.262528 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:44.262492 4926 generic.go:356] "Generic (PLEG): container finished" podID="fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6" containerID="1ed8738b729ed9cb467df3980329faf4ea7c9d91419c3df816a92260bcb2b615" exitCode=0 Oct 30 21:57:44.262957 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:44.262544 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5" event={"ID":"fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6","Type":"ContainerDied","Data":"1ed8738b729ed9cb467df3980329faf4ea7c9d91419c3df816a92260bcb2b615"} Oct 30 21:57:45.420918 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:45.420891 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5" Oct 30 21:57:45.558768 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:45.558674 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6-kube-api-access-gqdbj\" (UniqueName: \"kubernetes.io/projected/fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6-kube-api-access-gqdbj\") pod \"fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6\" (UID: \"fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6\") " Oct 30 21:57:45.558768 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:45.558763 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6-maas-api-openshift-ca\") pod \"fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6\" (UID: \"fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6\") " Oct 30 21:57:45.559017 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:45.558963 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6-maas-api-xks-ca\") pod \"fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6\" (UID: \"fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6\") " Oct 30 21:57:45.559328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:45.559298 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6-maas-api-openshift-ca" pod "fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6" (UID: "fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:45.559482 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:45.559462 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:45.561177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:45.561150 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6-maas-api-xks-ca" pod "fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6" (UID: "fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:45.561263 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:45.561237 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6-kube-api-access-gqdbj" pod "fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6" (UID: "fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6"). InnerVolumeSpecName "kube-api-access-gqdbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:45.660397 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:45.660350 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:45.660397 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:45.660390 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqdbj\" (UniqueName: \"kubernetes.io/projected/fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6-kube-api-access-gqdbj\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:46.280862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:46.280749 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5" Oct 30 21:57:46.280862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:46.280761 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5" event={"ID":"fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6","Type":"ContainerDied","Data":"85e23af8a9a7da720d28964306e6e400ba902fd0bc92904de4e7e02e50101531"} Oct 30 21:57:46.280862 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:46.280818 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85e23af8a9a7da720d28964306e6e400ba902fd0bc92904de4e7e02e50101531" Oct 30 21:57:54.507959 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.507909 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d70dc77b-77p7f"] Oct 30 21:57:54.508446 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.508430 4926 cpu_manager.go:414] "containerMap: removing container" podUID="fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6" containerName="revoke-keys" Oct 30 21:57:54.508491 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.508448 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6" containerName="revoke-keys" Oct 30 21:57:54.508552 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.508542 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6" containerName="revoke-keys" Oct 30 21:57:54.508589 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.508556 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="f40cfadc-67d6-467e-bced-347b09dd6462" containerName="istio-proxy" Oct 30 21:57:54.512109 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.512089 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70dc77b-77p7f" Oct 30 21:57:54.530320 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.530291 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd92aef-n5hh8"] Oct 30 21:57:54.533927 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.533907 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d70dc77b-77p7f"] Oct 30 21:57:54.534067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.534001 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd92aef-n5hh8" Oct 30 21:57:54.537739 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.537716 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:57:54.537943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.537922 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:57:54.538102 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.538026 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:57:54.538169 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.538118 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:57:54.563959 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.563922 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd13f33-da83-4b40-b2ed-102936728505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d70dc77b-77p7f\" (UID: \"1fd13f33-da83-4b40-b2ed-102936728505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70dc77b-77p7f" Oct 30 21:57:54.564306 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.564009 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4kfv\" (UniqueName: \"kubernetes.io/projected/1fd13f33-da83-4b40-b2ed-102936728505-kube-api-access-f4kfv\") pod \"maas-api-revoke-sub-d70dc77b-77p7f\" (UID: \"1fd13f33-da83-4b40-b2ed-102936728505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70dc77b-77p7f" Oct 30 21:57:54.564306 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.564085 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90bd49e9-63e3-4e33-a644-4441e33f5663-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdd92aef-n5hh8\" (UID: \"90bd49e9-63e3-4e33-a644-4441e33f5663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd92aef-n5hh8" Oct 30 21:57:54.564306 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.564232 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd13f33-da83-4b40-b2ed-102936728505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d70dc77b-77p7f\" (UID: \"1fd13f33-da83-4b40-b2ed-102936728505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70dc77b-77p7f" Oct 30 21:57:54.564306 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.564290 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90bd49e9-63e3-4e33-a644-4441e33f5663-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdd92aef-n5hh8\" (UID: \"90bd49e9-63e3-4e33-a644-4441e33f5663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd92aef-n5hh8" Oct 30 21:57:54.564494 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.564315 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96zlf\" (UniqueName: \"kubernetes.io/projected/90bd49e9-63e3-4e33-a644-4441e33f5663-kube-api-access-96zlf\") pod \"maas-api-revoke-sub-bdd92aef-n5hh8\" (UID: \"90bd49e9-63e3-4e33-a644-4441e33f5663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd92aef-n5hh8" Oct 30 21:57:54.571060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.570026 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd92aef-n5hh8"] Oct 30 21:57:54.665154 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.665115 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90bd49e9-63e3-4e33-a644-4441e33f5663-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdd92aef-n5hh8\" (UID: \"90bd49e9-63e3-4e33-a644-4441e33f5663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd92aef-n5hh8" Oct 30 21:57:54.665154 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.665155 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96zlf\" (UniqueName: \"kubernetes.io/projected/90bd49e9-63e3-4e33-a644-4441e33f5663-kube-api-access-96zlf\") pod \"maas-api-revoke-sub-bdd92aef-n5hh8\" (UID: \"90bd49e9-63e3-4e33-a644-4441e33f5663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd92aef-n5hh8" Oct 30 21:57:54.665396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.665239 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd13f33-da83-4b40-b2ed-102936728505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d70dc77b-77p7f\" (UID: \"1fd13f33-da83-4b40-b2ed-102936728505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70dc77b-77p7f" Oct 30 21:57:54.665396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.665287 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4kfv\" (UniqueName: \"kubernetes.io/projected/1fd13f33-da83-4b40-b2ed-102936728505-kube-api-access-f4kfv\") pod \"maas-api-revoke-sub-d70dc77b-77p7f\" (UID: \"1fd13f33-da83-4b40-b2ed-102936728505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70dc77b-77p7f" Oct 30 21:57:54.665396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.665333 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90bd49e9-63e3-4e33-a644-4441e33f5663-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdd92aef-n5hh8\" (UID: \"90bd49e9-63e3-4e33-a644-4441e33f5663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd92aef-n5hh8" Oct 30 21:57:54.665396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.665377 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd13f33-da83-4b40-b2ed-102936728505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d70dc77b-77p7f\" (UID: \"1fd13f33-da83-4b40-b2ed-102936728505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70dc77b-77p7f" Oct 30 21:57:54.666129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.666100 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90bd49e9-63e3-4e33-a644-4441e33f5663-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdd92aef-n5hh8\" (UID: \"90bd49e9-63e3-4e33-a644-4441e33f5663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd92aef-n5hh8" Oct 30 21:57:54.666129 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.666118 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd13f33-da83-4b40-b2ed-102936728505-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d70dc77b-77p7f\" (UID: \"1fd13f33-da83-4b40-b2ed-102936728505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70dc77b-77p7f" Oct 30 21:57:54.667618 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.667595 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd13f33-da83-4b40-b2ed-102936728505-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d70dc77b-77p7f\" (UID: \"1fd13f33-da83-4b40-b2ed-102936728505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70dc77b-77p7f" Oct 30 21:57:54.667725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.667708 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90bd49e9-63e3-4e33-a644-4441e33f5663-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdd92aef-n5hh8\" (UID: \"90bd49e9-63e3-4e33-a644-4441e33f5663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd92aef-n5hh8" Oct 30 21:57:54.688295 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.688260 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4kfv\" (UniqueName: \"kubernetes.io/projected/1fd13f33-da83-4b40-b2ed-102936728505-kube-api-access-f4kfv\") pod \"maas-api-revoke-sub-d70dc77b-77p7f\" (UID: \"1fd13f33-da83-4b40-b2ed-102936728505\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70dc77b-77p7f" Oct 30 21:57:54.689420 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.689370 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96zlf\" (UniqueName: \"kubernetes.io/projected/90bd49e9-63e3-4e33-a644-4441e33f5663-kube-api-access-96zlf\") pod \"maas-api-revoke-sub-bdd92aef-n5hh8\" (UID: \"90bd49e9-63e3-4e33-a644-4441e33f5663\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd92aef-n5hh8" Oct 30 21:57:54.738679 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.738638 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:57:54.828207 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.828088 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70dc77b-77p7f" Oct 30 21:57:54.849255 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:54.849227 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd92aef-n5hh8" Oct 30 21:57:55.136992 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:55.136960 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d70dc77b-77p7f"] Oct 30 21:57:55.138111 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:57:55.138085 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd13f33_da83_4b40_b2ed_102936728505.slice/crio-2e8b00e6bd5a61dfe484b1e3cc5bfcdc51f2cb0250b4c8bf6121c69a6732f250 WatchSource:0}: Error finding container 2e8b00e6bd5a61dfe484b1e3cc5bfcdc51f2cb0250b4c8bf6121c69a6732f250: Status 404 returned error can't find the container with id 2e8b00e6bd5a61dfe484b1e3cc5bfcdc51f2cb0250b4c8bf6121c69a6732f250 Oct 30 21:57:55.166422 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:55.166394 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd92aef-n5hh8"] Oct 30 21:57:55.170347 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:57:55.170320 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90bd49e9_63e3_4e33_a644_4441e33f5663.slice/crio-eabff73eea2c95ddb46336a85f35a977982a7401228a22d6bbab9461e7085120 WatchSource:0}: Error finding container eabff73eea2c95ddb46336a85f35a977982a7401228a22d6bbab9461e7085120: Status 404 returned error can't find the container with id eabff73eea2c95ddb46336a85f35a977982a7401228a22d6bbab9461e7085120 Oct 30 21:57:55.359434 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:55.359393 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70dc77b-77p7f" event={"ID":"1fd13f33-da83-4b40-b2ed-102936728505","Type":"ContainerStarted","Data":"2e8b00e6bd5a61dfe484b1e3cc5bfcdc51f2cb0250b4c8bf6121c69a6732f250"} Oct 30 21:57:55.360860 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:55.360825 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd92aef-n5hh8" event={"ID":"90bd49e9-63e3-4e33-a644-4441e33f5663","Type":"ContainerStarted","Data":"eabff73eea2c95ddb46336a85f35a977982a7401228a22d6bbab9461e7085120"} Oct 30 21:57:56.370434 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:56.370399 4926 generic.go:356] "Generic (PLEG): container finished" podID="1fd13f33-da83-4b40-b2ed-102936728505" containerID="198b927f1603c08c3d75b5fc6960d241b966ecf62ecf137910ffa216202e8fd5" exitCode=0 Oct 30 21:57:56.370942 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:56.370491 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70dc77b-77p7f" event={"ID":"1fd13f33-da83-4b40-b2ed-102936728505","Type":"ContainerDied","Data":"198b927f1603c08c3d75b5fc6960d241b966ecf62ecf137910ffa216202e8fd5"} Oct 30 21:57:56.372746 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:56.372722 4926 generic.go:356] "Generic (PLEG): container finished" podID="90bd49e9-63e3-4e33-a644-4441e33f5663" containerID="5b155f3fd39708cf9da7e27da7f0a04667550fcc11127226ce70a987a7e15684" exitCode=0 Oct 30 21:57:56.372895 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:56.372808 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd92aef-n5hh8" event={"ID":"90bd49e9-63e3-4e33-a644-4441e33f5663","Type":"ContainerDied","Data":"5b155f3fd39708cf9da7e27da7f0a04667550fcc11127226ce70a987a7e15684"} Oct 30 21:57:56.886650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:56.886612 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" containerID="cri-o://ae809aacc93f33f978e99d43178b4be1a8d7c3b3648d5df320b972368e0e0983" gracePeriod=45 Oct 30 21:57:56.901479 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:56.901438 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" containerID="cri-o://421dc69ae5bde6099528b1c50477a192de6575bccfc5aa44cbb5f43ac536bb33" gracePeriod=45 Oct 30 21:57:56.907951 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:56.907922 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": dial tcp 10.132.0.84:8000: connect: connection refused" Oct 30 21:57:56.921722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:56.921688 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" containerID="cri-o://3f52ba3f3a4e0075151a30ea90011884e91592b976da0d103bdbdc0db2dcc992" gracePeriod=45 Oct 30 21:57:57.217141 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.217112 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:57:57.220771 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.220741 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:57:57.249724 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.249695 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:57:57.293526 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.293484 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-tmp-dir\") pod \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " Oct 30 21:57:57.293725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.293530 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af79cf23-fc31-4666-842f-5af27a17cb02-kube-api-access-5jn7l\" (UniqueName: \"kubernetes.io/projected/af79cf23-fc31-4666-842f-5af27a17cb02-kube-api-access-5jn7l\") pod \"af79cf23-fc31-4666-842f-5af27a17cb02\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " Oct 30 21:57:57.293725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.293578 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-dshm\") pod \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " Oct 30 21:57:57.293725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.293654 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-home\" (UniqueName: \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-home\") pod \"af79cf23-fc31-4666-842f-5af27a17cb02\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " Oct 30 21:57:57.293725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.293691 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-model-cache\") pod \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " Oct 30 21:57:57.293947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.293729 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-model-cache\") pod \"af79cf23-fc31-4666-842f-5af27a17cb02\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " Oct 30 21:57:57.293947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.293807 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-home\" (UniqueName: \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-home\") pod \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " Oct 30 21:57:57.293947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.293810 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-tmp-dir" pod "e8733709-97b3-4b3a-85cd-f47c62a9fc71" (UID: "e8733709-97b3-4b3a-85cd-f47c62a9fc71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:57.293947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.293833 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-tmp-dir\") pod \"af79cf23-fc31-4666-842f-5af27a17cb02\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " Oct 30 21:57:57.293947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.293867 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-home\" (UniqueName: \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-home\") pod \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " Oct 30 21:57:57.293947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.293900 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-dshm\") pod \"af79cf23-fc31-4666-842f-5af27a17cb02\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " Oct 30 21:57:57.293947 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.293938 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21cab79c-3466-4600-983d-d8ea4f9dc0d9-kube-api-access-mffc6\" (UniqueName: \"kubernetes.io/projected/21cab79c-3466-4600-983d-d8ea4f9dc0d9-kube-api-access-mffc6\") pod \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " Oct 30 21:57:57.294335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.293979 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-model-cache\") pod \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " Oct 30 21:57:57.294335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.294007 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-dshm\") pod \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " Oct 30 21:57:57.294335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.294019 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-model-cache" pod "21cab79c-3466-4600-983d-d8ea4f9dc0d9" (UID: "21cab79c-3466-4600-983d-d8ea4f9dc0d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:57.294335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.294055 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-tmp-dir\") pod \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " Oct 30 21:57:57.294335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.294086 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af79cf23-fc31-4666-842f-5af27a17cb02-tls-certs\" (UniqueName: \"kubernetes.io/secret/af79cf23-fc31-4666-842f-5af27a17cb02-tls-certs\") pod \"af79cf23-fc31-4666-842f-5af27a17cb02\" (UID: \"af79cf23-fc31-4666-842f-5af27a17cb02\") " Oct 30 21:57:57.294335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.294142 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21cab79c-3466-4600-983d-d8ea4f9dc0d9-tls-certs\" (UniqueName: \"kubernetes.io/secret/21cab79c-3466-4600-983d-d8ea4f9dc0d9-tls-certs\") pod \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\" (UID: \"21cab79c-3466-4600-983d-d8ea4f9dc0d9\") " Oct 30 21:57:57.294335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.294114 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-home" pod "21cab79c-3466-4600-983d-d8ea4f9dc0d9" (UID: "21cab79c-3466-4600-983d-d8ea4f9dc0d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:57.294335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.294180 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8733709-97b3-4b3a-85cd-f47c62a9fc71-tls-certs\" (UniqueName: \"kubernetes.io/secret/e8733709-97b3-4b3a-85cd-f47c62a9fc71-tls-certs\") pod \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " Oct 30 21:57:57.294335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.294221 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8733709-97b3-4b3a-85cd-f47c62a9fc71-kube-api-access-qq9nk\" (UniqueName: \"kubernetes.io/projected/e8733709-97b3-4b3a-85cd-f47c62a9fc71-kube-api-access-qq9nk\") pod \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\" (UID: \"e8733709-97b3-4b3a-85cd-f47c62a9fc71\") " Oct 30 21:57:57.294335 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.294228 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-home" pod "af79cf23-fc31-4666-842f-5af27a17cb02" (UID: "af79cf23-fc31-4666-842f-5af27a17cb02"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:57.294855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.294351 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-model-cache" pod "af79cf23-fc31-4666-842f-5af27a17cb02" (UID: "af79cf23-fc31-4666-842f-5af27a17cb02"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:57.294855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.294601 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-model-cache" pod "e8733709-97b3-4b3a-85cd-f47c62a9fc71" (UID: "e8733709-97b3-4b3a-85cd-f47c62a9fc71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:57.294855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.294675 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.294855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.294699 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.294855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.294716 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.294855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.294730 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.294855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.294743 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.295246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.294942 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-home" pod "e8733709-97b3-4b3a-85cd-f47c62a9fc71" (UID: "e8733709-97b3-4b3a-85cd-f47c62a9fc71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:57.295303 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.295237 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-tmp-dir" pod "af79cf23-fc31-4666-842f-5af27a17cb02" (UID: "af79cf23-fc31-4666-842f-5af27a17cb02"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:57.296782 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.296748 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af79cf23-fc31-4666-842f-5af27a17cb02-kube-api-access-5jn7l" pod "af79cf23-fc31-4666-842f-5af27a17cb02" (UID: "af79cf23-fc31-4666-842f-5af27a17cb02"). InnerVolumeSpecName "kube-api-access-5jn7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:57.296931 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.296880 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8733709-97b3-4b3a-85cd-f47c62a9fc71-kube-api-access-qq9nk" pod "e8733709-97b3-4b3a-85cd-f47c62a9fc71" (UID: "e8733709-97b3-4b3a-85cd-f47c62a9fc71"). InnerVolumeSpecName "kube-api-access-qq9nk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:57.297136 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.297102 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-tmp-dir" pod "21cab79c-3466-4600-983d-d8ea4f9dc0d9" (UID: "21cab79c-3466-4600-983d-d8ea4f9dc0d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:57.297484 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.297337 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8733709-97b3-4b3a-85cd-f47c62a9fc71-tls-certs" pod "e8733709-97b3-4b3a-85cd-f47c62a9fc71" (UID: "e8733709-97b3-4b3a-85cd-f47c62a9fc71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:57.299281 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.298588 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af79cf23-fc31-4666-842f-5af27a17cb02-tls-certs" pod "af79cf23-fc31-4666-842f-5af27a17cb02" (UID: "af79cf23-fc31-4666-842f-5af27a17cb02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:57.299281 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.298597 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-dshm" pod "21cab79c-3466-4600-983d-d8ea4f9dc0d9" (UID: "21cab79c-3466-4600-983d-d8ea4f9dc0d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:57.299281 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.298752 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-dshm" pod "af79cf23-fc31-4666-842f-5af27a17cb02" (UID: "af79cf23-fc31-4666-842f-5af27a17cb02"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:57.300644 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.299834 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-dshm" pod "e8733709-97b3-4b3a-85cd-f47c62a9fc71" (UID: "e8733709-97b3-4b3a-85cd-f47c62a9fc71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:57.300644 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.299895 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21cab79c-3466-4600-983d-d8ea4f9dc0d9-kube-api-access-mffc6" pod "21cab79c-3466-4600-983d-d8ea4f9dc0d9" (UID: "21cab79c-3466-4600-983d-d8ea4f9dc0d9"). InnerVolumeSpecName "kube-api-access-mffc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:57.300644 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.299959 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21cab79c-3466-4600-983d-d8ea4f9dc0d9-tls-certs" pod "21cab79c-3466-4600-983d-d8ea4f9dc0d9" (UID: "21cab79c-3466-4600-983d-d8ea4f9dc0d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:57.391225 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.391184 4926 generic.go:356] "Generic (PLEG): container finished" podID="af79cf23-fc31-4666-842f-5af27a17cb02" containerID="421dc69ae5bde6099528b1c50477a192de6575bccfc5aa44cbb5f43ac536bb33" exitCode=0 Oct 30 21:57:57.391747 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.391237 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" Oct 30 21:57:57.391747 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.391288 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" event={"ID":"af79cf23-fc31-4666-842f-5af27a17cb02","Type":"ContainerDied","Data":"421dc69ae5bde6099528b1c50477a192de6575bccfc5aa44cbb5f43ac536bb33"} Oct 30 21:57:57.391747 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.391360 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr" event={"ID":"af79cf23-fc31-4666-842f-5af27a17cb02","Type":"ContainerDied","Data":"263c3d96939cead756b31c57135b80fe89876916f984286623b5878cc01e6f40"} Oct 30 21:57:57.391747 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.391390 4926 scope.go:122] "RemoveContainer" containerID="421dc69ae5bde6099528b1c50477a192de6575bccfc5aa44cbb5f43ac536bb33" Oct 30 21:57:57.395204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.394856 4926 generic.go:356] "Generic (PLEG): container finished" podID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerID="3f52ba3f3a4e0075151a30ea90011884e91592b976da0d103bdbdc0db2dcc992" exitCode=0 Oct 30 21:57:57.395204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.394982 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" Oct 30 21:57:57.395204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.395131 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" event={"ID":"e8733709-97b3-4b3a-85cd-f47c62a9fc71","Type":"ContainerDied","Data":"3f52ba3f3a4e0075151a30ea90011884e91592b976da0d103bdbdc0db2dcc992"} Oct 30 21:57:57.395204 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.395165 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" event={"ID":"e8733709-97b3-4b3a-85cd-f47c62a9fc71","Type":"ContainerDied","Data":"6886b0055a88088c409394f2727236b7d4da4259fcb0811c5fd7c03ad4a17fff"} Oct 30 21:57:57.395632 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.395598 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.395632 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.395631 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.395787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.395647 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af79cf23-fc31-4666-842f-5af27a17cb02-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.395787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.395663 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mffc6\" (UniqueName: \"kubernetes.io/projected/21cab79c-3466-4600-983d-d8ea4f9dc0d9-kube-api-access-mffc6\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.395787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.395679 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.395787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.395693 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.395787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.395706 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21cab79c-3466-4600-983d-d8ea4f9dc0d9-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.395787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.395719 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af79cf23-fc31-4666-842f-5af27a17cb02-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.395787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.395732 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21cab79c-3466-4600-983d-d8ea4f9dc0d9-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.395787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.395747 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e8733709-97b3-4b3a-85cd-f47c62a9fc71-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.395787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.395761 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq9nk\" (UniqueName: \"kubernetes.io/projected/e8733709-97b3-4b3a-85cd-f47c62a9fc71-kube-api-access-qq9nk\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.395787 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.395781 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jn7l\" (UniqueName: \"kubernetes.io/projected/af79cf23-fc31-4666-842f-5af27a17cb02-kube-api-access-5jn7l\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.396355 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.395796 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e8733709-97b3-4b3a-85cd-f47c62a9fc71-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.404003 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.403973 4926 generic.go:356] "Generic (PLEG): container finished" podID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerID="ae809aacc93f33f978e99d43178b4be1a8d7c3b3648d5df320b972368e0e0983" exitCode=0 Oct 30 21:57:57.404187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.404019 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" Oct 30 21:57:57.404187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.404094 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" event={"ID":"21cab79c-3466-4600-983d-d8ea4f9dc0d9","Type":"ContainerDied","Data":"ae809aacc93f33f978e99d43178b4be1a8d7c3b3648d5df320b972368e0e0983"} Oct 30 21:57:57.404187 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.404125 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps" event={"ID":"21cab79c-3466-4600-983d-d8ea4f9dc0d9","Type":"ContainerDied","Data":"9a9fcd08ff5bbed67542d69731a85f4189b7ee27389b7af7e98569df8e7468c8"} Oct 30 21:57:57.410408 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.410384 4926 scope.go:122] "RemoveContainer" containerID="421dc69ae5bde6099528b1c50477a192de6575bccfc5aa44cbb5f43ac536bb33" Oct 30 21:57:57.410803 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:57:57.410771 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"421dc69ae5bde6099528b1c50477a192de6575bccfc5aa44cbb5f43ac536bb33\": container with ID starting with 421dc69ae5bde6099528b1c50477a192de6575bccfc5aa44cbb5f43ac536bb33 not found: ID does not exist" containerID="421dc69ae5bde6099528b1c50477a192de6575bccfc5aa44cbb5f43ac536bb33" Oct 30 21:57:57.410921 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.410815 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"421dc69ae5bde6099528b1c50477a192de6575bccfc5aa44cbb5f43ac536bb33"} err="failed to get container status \"421dc69ae5bde6099528b1c50477a192de6575bccfc5aa44cbb5f43ac536bb33\": rpc error: code = NotFound desc = could not find container \"421dc69ae5bde6099528b1c50477a192de6575bccfc5aa44cbb5f43ac536bb33\": container with ID starting with 421dc69ae5bde6099528b1c50477a192de6575bccfc5aa44cbb5f43ac536bb33 not found: ID does not exist" Oct 30 21:57:57.410921 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.410838 4926 scope.go:122] "RemoveContainer" containerID="3f52ba3f3a4e0075151a30ea90011884e91592b976da0d103bdbdc0db2dcc992" Oct 30 21:57:57.427417 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.427393 4926 scope.go:122] "RemoveContainer" containerID="3f52ba3f3a4e0075151a30ea90011884e91592b976da0d103bdbdc0db2dcc992" Oct 30 21:57:57.428263 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:57:57.427825 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f52ba3f3a4e0075151a30ea90011884e91592b976da0d103bdbdc0db2dcc992\": container with ID starting with 3f52ba3f3a4e0075151a30ea90011884e91592b976da0d103bdbdc0db2dcc992 not found: ID does not exist" containerID="3f52ba3f3a4e0075151a30ea90011884e91592b976da0d103bdbdc0db2dcc992" Oct 30 21:57:57.428263 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.427868 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f52ba3f3a4e0075151a30ea90011884e91592b976da0d103bdbdc0db2dcc992"} err="failed to get container status \"3f52ba3f3a4e0075151a30ea90011884e91592b976da0d103bdbdc0db2dcc992\": rpc error: code = NotFound desc = could not find container \"3f52ba3f3a4e0075151a30ea90011884e91592b976da0d103bdbdc0db2dcc992\": container with ID starting with 3f52ba3f3a4e0075151a30ea90011884e91592b976da0d103bdbdc0db2dcc992 not found: ID does not exist" Oct 30 21:57:57.428263 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.427894 4926 scope.go:122] "RemoveContainer" containerID="ae809aacc93f33f978e99d43178b4be1a8d7c3b3648d5df320b972368e0e0983" Oct 30 21:57:57.443021 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.442997 4926 scope.go:122] "RemoveContainer" containerID="ae809aacc93f33f978e99d43178b4be1a8d7c3b3648d5df320b972368e0e0983" Oct 30 21:57:57.443508 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:57:57.443483 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae809aacc93f33f978e99d43178b4be1a8d7c3b3648d5df320b972368e0e0983\": container with ID starting with ae809aacc93f33f978e99d43178b4be1a8d7c3b3648d5df320b972368e0e0983 not found: ID does not exist" containerID="ae809aacc93f33f978e99d43178b4be1a8d7c3b3648d5df320b972368e0e0983" Oct 30 21:57:57.443583 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.443518 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae809aacc93f33f978e99d43178b4be1a8d7c3b3648d5df320b972368e0e0983"} err="failed to get container status \"ae809aacc93f33f978e99d43178b4be1a8d7c3b3648d5df320b972368e0e0983\": rpc error: code = NotFound desc = could not find container \"ae809aacc93f33f978e99d43178b4be1a8d7c3b3648d5df320b972368e0e0983\": container with ID starting with ae809aacc93f33f978e99d43178b4be1a8d7c3b3648d5df320b972368e0e0983 not found: ID does not exist" Oct 30 21:57:57.502326 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.502292 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr"] Oct 30 21:57:57.517430 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.517399 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-43c070/distinct-w5-43c070-kserve-79577d4b57-f5hhr"] Oct 30 21:57:57.552555 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.552519 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx"] Oct 30 21:57:57.564404 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.564375 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx"] Oct 30 21:57:57.569588 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.569566 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd92aef-n5hh8" Oct 30 21:57:57.572917 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.572896 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70dc77b-77p7f" Oct 30 21:57:57.601067 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.601011 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps"] Oct 30 21:57:57.604328 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.604299 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-43c070/distinct-2-w5-43c070-kserve-64f96c689d-nthps"] Oct 30 21:57:57.698558 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.698514 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fd13f33-da83-4b40-b2ed-102936728505-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd13f33-da83-4b40-b2ed-102936728505-maas-api-openshift-ca\") pod \"1fd13f33-da83-4b40-b2ed-102936728505\" (UID: \"1fd13f33-da83-4b40-b2ed-102936728505\") " Oct 30 21:57:57.698768 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.698585 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd13f33-da83-4b40-b2ed-102936728505-kube-api-access-f4kfv\" (UniqueName: \"kubernetes.io/projected/1fd13f33-da83-4b40-b2ed-102936728505-kube-api-access-f4kfv\") pod \"1fd13f33-da83-4b40-b2ed-102936728505\" (UID: \"1fd13f33-da83-4b40-b2ed-102936728505\") " Oct 30 21:57:57.698768 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.698658 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fd13f33-da83-4b40-b2ed-102936728505-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd13f33-da83-4b40-b2ed-102936728505-maas-api-xks-ca\") pod \"1fd13f33-da83-4b40-b2ed-102936728505\" (UID: \"1fd13f33-da83-4b40-b2ed-102936728505\") " Oct 30 21:57:57.698768 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.698736 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90bd49e9-63e3-4e33-a644-4441e33f5663-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90bd49e9-63e3-4e33-a644-4441e33f5663-maas-api-openshift-ca\") pod \"90bd49e9-63e3-4e33-a644-4441e33f5663\" (UID: \"90bd49e9-63e3-4e33-a644-4441e33f5663\") " Oct 30 21:57:57.698938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.698792 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90bd49e9-63e3-4e33-a644-4441e33f5663-kube-api-access-96zlf\" (UniqueName: \"kubernetes.io/projected/90bd49e9-63e3-4e33-a644-4441e33f5663-kube-api-access-96zlf\") pod \"90bd49e9-63e3-4e33-a644-4441e33f5663\" (UID: \"90bd49e9-63e3-4e33-a644-4441e33f5663\") " Oct 30 21:57:57.698938 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.698821 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90bd49e9-63e3-4e33-a644-4441e33f5663-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90bd49e9-63e3-4e33-a644-4441e33f5663-maas-api-xks-ca\") pod \"90bd49e9-63e3-4e33-a644-4441e33f5663\" (UID: \"90bd49e9-63e3-4e33-a644-4441e33f5663\") " Oct 30 21:57:57.699060 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.699018 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd13f33-da83-4b40-b2ed-102936728505-maas-api-openshift-ca" pod "1fd13f33-da83-4b40-b2ed-102936728505" (UID: "1fd13f33-da83-4b40-b2ed-102936728505"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:57.699265 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.699239 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fd13f33-da83-4b40-b2ed-102936728505-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.699441 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.699408 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90bd49e9-63e3-4e33-a644-4441e33f5663-maas-api-openshift-ca" pod "90bd49e9-63e3-4e33-a644-4441e33f5663" (UID: "90bd49e9-63e3-4e33-a644-4441e33f5663"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:57:57.701503 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.701473 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fd13f33-da83-4b40-b2ed-102936728505-maas-api-xks-ca" pod "1fd13f33-da83-4b40-b2ed-102936728505" (UID: "1fd13f33-da83-4b40-b2ed-102936728505"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:57.701671 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.701648 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd13f33-da83-4b40-b2ed-102936728505-kube-api-access-f4kfv" pod "1fd13f33-da83-4b40-b2ed-102936728505" (UID: "1fd13f33-da83-4b40-b2ed-102936728505"). InnerVolumeSpecName "kube-api-access-f4kfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:57.701722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.701674 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90bd49e9-63e3-4e33-a644-4441e33f5663-kube-api-access-96zlf" pod "90bd49e9-63e3-4e33-a644-4441e33f5663" (UID: "90bd49e9-63e3-4e33-a644-4441e33f5663"). InnerVolumeSpecName "kube-api-access-96zlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:57.701722 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.701654 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90bd49e9-63e3-4e33-a644-4441e33f5663-maas-api-xks-ca" pod "90bd49e9-63e3-4e33-a644-4441e33f5663" (UID: "90bd49e9-63e3-4e33-a644-4441e33f5663"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:57.800767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.800727 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4kfv\" (UniqueName: \"kubernetes.io/projected/1fd13f33-da83-4b40-b2ed-102936728505-kube-api-access-f4kfv\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.800767 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.800764 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fd13f33-da83-4b40-b2ed-102936728505-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.801068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.800779 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90bd49e9-63e3-4e33-a644-4441e33f5663-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.801068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.800792 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96zlf\" (UniqueName: \"kubernetes.io/projected/90bd49e9-63e3-4e33-a644-4441e33f5663-kube-api-access-96zlf\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:57.801068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:57.800805 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90bd49e9-63e3-4e33-a644-4441e33f5663-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:57:58.156189 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:58.156093 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-43c070/simulator-w5-43c070-kserve-f858d6fd7-z27jx" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 30 21:57:58.417387 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:58.417354 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70dc77b-77p7f" Oct 30 21:57:58.417821 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:58.417360 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d70dc77b-77p7f" event={"ID":"1fd13f33-da83-4b40-b2ed-102936728505","Type":"ContainerDied","Data":"2e8b00e6bd5a61dfe484b1e3cc5bfcdc51f2cb0250b4c8bf6121c69a6732f250"} Oct 30 21:57:58.417821 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:58.417496 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e8b00e6bd5a61dfe484b1e3cc5bfcdc51f2cb0250b4c8bf6121c69a6732f250" Oct 30 21:57:58.420249 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:58.420222 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd92aef-n5hh8" Oct 30 21:57:58.420395 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:58.420268 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdd92aef-n5hh8" event={"ID":"90bd49e9-63e3-4e33-a644-4441e33f5663","Type":"ContainerDied","Data":"eabff73eea2c95ddb46336a85f35a977982a7401228a22d6bbab9461e7085120"} Oct 30 21:57:58.420395 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:58.420316 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eabff73eea2c95ddb46336a85f35a977982a7401228a22d6bbab9461e7085120" Oct 30 21:57:58.484295 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:58.484254 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5"] Oct 30 21:57:58.491664 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:58.491627 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-908469-pqkr5"] Oct 30 21:57:59.015934 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:59.015899 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" path="/var/lib/kubelet/pods/21cab79c-3466-4600-983d-d8ea4f9dc0d9/volumes" Oct 30 21:57:59.016362 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:59.016348 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" path="/var/lib/kubelet/pods/af79cf23-fc31-4666-842f-5af27a17cb02/volumes" Oct 30 21:57:59.016754 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:59.016741 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" path="/var/lib/kubelet/pods/e8733709-97b3-4b3a-85cd-f47c62a9fc71/volumes" Oct 30 21:57:59.017448 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:57:59.017433 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6" path="/var/lib/kubelet/pods/fc35cd74-d73d-4a73-b1bb-07f8e51fe4d6/volumes" Oct 30 21:58:06.135997 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.135961 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg"] Oct 30 21:58:06.136503 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.136279 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" podUID="73a624ed-4a10-48c0-9a7a-002ebb6114df" containerName="istio-proxy" containerID="cri-o://7f92220df6c2c9884681329f95c7a94de88e08fda0c2be135bca9985d8cdfab5" gracePeriod=30 Oct 30 21:58:06.650621 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.650582 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:58:06.674471 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.674429 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz"] Oct 30 21:58:06.674964 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.674941 4926 cpu_manager.go:414] "containerMap: removing container" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" Oct 30 21:58:06.675082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.674965 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" Oct 30 21:58:06.675082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.675005 4926 cpu_manager.go:414] "containerMap: removing container" podUID="1fd13f33-da83-4b40-b2ed-102936728505" containerName="revoke-keys" Oct 30 21:58:06.675082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.675014 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd13f33-da83-4b40-b2ed-102936728505" containerName="revoke-keys" Oct 30 21:58:06.675082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.675057 4926 cpu_manager.go:414] "containerMap: removing container" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" Oct 30 21:58:06.675082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.675065 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" Oct 30 21:58:06.675082 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.675083 4926 cpu_manager.go:414] "containerMap: removing container" podUID="90bd49e9-63e3-4e33-a644-4441e33f5663" containerName="revoke-keys" Oct 30 21:58:06.675395 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.675091 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90bd49e9-63e3-4e33-a644-4441e33f5663" containerName="revoke-keys" Oct 30 21:58:06.675395 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.675101 4926 cpu_manager.go:414] "containerMap: removing container" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" Oct 30 21:58:06.675395 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.675109 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" Oct 30 21:58:06.675395 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.675231 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8733709-97b3-4b3a-85cd-f47c62a9fc71" containerName="main" Oct 30 21:58:06.675395 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.675252 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd13f33-da83-4b40-b2ed-102936728505" containerName="revoke-keys" Oct 30 21:58:06.675395 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.675264 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="af79cf23-fc31-4666-842f-5af27a17cb02" containerName="main" Oct 30 21:58:06.675395 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.675275 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="90bd49e9-63e3-4e33-a644-4441e33f5663" containerName="revoke-keys" Oct 30 21:58:06.675395 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.675283 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="21cab79c-3466-4600-983d-d8ea4f9dc0d9" containerName="main" Oct 30 21:58:06.679952 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.679927 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz" Oct 30 21:58:06.696618 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.696584 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz"] Oct 30 21:58:06.698196 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.698169 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:58:06.698196 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.698172 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:58:06.788302 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.788262 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:58:06.788501 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.788329 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:58:06.797846 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.797806 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de4a3ad-a331-4ef2-b1ec-e75147a26601-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz\" (UID: \"4de4a3ad-a331-4ef2-b1ec-e75147a26601\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz" Oct 30 21:58:06.798079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.797880 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsjkb\" (UniqueName: \"kubernetes.io/projected/4de4a3ad-a331-4ef2-b1ec-e75147a26601-kube-api-access-bsjkb\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz\" (UID: \"4de4a3ad-a331-4ef2-b1ec-e75147a26601\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz" Oct 30 21:58:06.798079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.797990 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de4a3ad-a331-4ef2-b1ec-e75147a26601-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz\" (UID: \"4de4a3ad-a331-4ef2-b1ec-e75147a26601\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz" Oct 30 21:58:06.899380 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.899340 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de4a3ad-a331-4ef2-b1ec-e75147a26601-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz\" (UID: \"4de4a3ad-a331-4ef2-b1ec-e75147a26601\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz" Oct 30 21:58:06.899574 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.899401 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsjkb\" (UniqueName: \"kubernetes.io/projected/4de4a3ad-a331-4ef2-b1ec-e75147a26601-kube-api-access-bsjkb\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz\" (UID: \"4de4a3ad-a331-4ef2-b1ec-e75147a26601\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz" Oct 30 21:58:06.899574 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.899422 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de4a3ad-a331-4ef2-b1ec-e75147a26601-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz\" (UID: \"4de4a3ad-a331-4ef2-b1ec-e75147a26601\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz" Oct 30 21:58:06.900021 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.900001 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de4a3ad-a331-4ef2-b1ec-e75147a26601-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz\" (UID: \"4de4a3ad-a331-4ef2-b1ec-e75147a26601\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz" Oct 30 21:58:06.902094 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.902016 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de4a3ad-a331-4ef2-b1ec-e75147a26601-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz\" (UID: \"4de4a3ad-a331-4ef2-b1ec-e75147a26601\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz" Oct 30 21:58:06.918192 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.918162 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsjkb\" (UniqueName: \"kubernetes.io/projected/4de4a3ad-a331-4ef2-b1ec-e75147a26601-kube-api-access-bsjkb\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz\" (UID: \"4de4a3ad-a331-4ef2-b1ec-e75147a26601\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz" Oct 30 21:58:06.995123 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:06.995085 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz" Oct 30 21:58:07.305817 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:07.305786 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz"] Oct 30 21:58:07.308349 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:58:07.308322 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4de4a3ad_a331_4ef2_b1ec_e75147a26601.slice/crio-692788d51c12d55e43553dae1b98ba191080d1c5a619f9153da20668d9b1bc02 WatchSource:0}: Error finding container 692788d51c12d55e43553dae1b98ba191080d1c5a619f9153da20668d9b1bc02: Status 404 returned error can't find the container with id 692788d51c12d55e43553dae1b98ba191080d1c5a619f9153da20668d9b1bc02 Oct 30 21:58:07.486018 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:58:07.485979 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:58:07.501997 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:07.501964 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz" event={"ID":"4de4a3ad-a331-4ef2-b1ec-e75147a26601","Type":"ContainerStarted","Data":"692788d51c12d55e43553dae1b98ba191080d1c5a619f9153da20668d9b1bc02"} Oct 30 21:58:08.514884 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:08.514785 4926 generic.go:356] "Generic (PLEG): container finished" podID="4de4a3ad-a331-4ef2-b1ec-e75147a26601" containerID="4c9cffb1cade8459d7b90c157b7edf61bf717db69b17056c447bc694d25859b7" exitCode=0 Oct 30 21:58:08.515338 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:08.514864 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz" event={"ID":"4de4a3ad-a331-4ef2-b1ec-e75147a26601","Type":"ContainerDied","Data":"4c9cffb1cade8459d7b90c157b7edf61bf717db69b17056c447bc694d25859b7"} Oct 30 21:58:09.672884 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:09.672852 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz" Oct 30 21:58:09.831597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:09.831515 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4de4a3ad-a331-4ef2-b1ec-e75147a26601-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de4a3ad-a331-4ef2-b1ec-e75147a26601-maas-api-openshift-ca\") pod \"4de4a3ad-a331-4ef2-b1ec-e75147a26601\" (UID: \"4de4a3ad-a331-4ef2-b1ec-e75147a26601\") " Oct 30 21:58:09.831597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:09.831568 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4de4a3ad-a331-4ef2-b1ec-e75147a26601-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de4a3ad-a331-4ef2-b1ec-e75147a26601-maas-api-xks-ca\") pod \"4de4a3ad-a331-4ef2-b1ec-e75147a26601\" (UID: \"4de4a3ad-a331-4ef2-b1ec-e75147a26601\") " Oct 30 21:58:09.831855 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:09.831672 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4de4a3ad-a331-4ef2-b1ec-e75147a26601-kube-api-access-bsjkb\" (UniqueName: \"kubernetes.io/projected/4de4a3ad-a331-4ef2-b1ec-e75147a26601-kube-api-access-bsjkb\") pod \"4de4a3ad-a331-4ef2-b1ec-e75147a26601\" (UID: \"4de4a3ad-a331-4ef2-b1ec-e75147a26601\") " Oct 30 21:58:09.831935 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:09.831890 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4de4a3ad-a331-4ef2-b1ec-e75147a26601-maas-api-openshift-ca" pod "4de4a3ad-a331-4ef2-b1ec-e75147a26601" (UID: "4de4a3ad-a331-4ef2-b1ec-e75147a26601"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:09.832118 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:09.832096 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4de4a3ad-a331-4ef2-b1ec-e75147a26601-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:09.833779 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:09.833752 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4de4a3ad-a331-4ef2-b1ec-e75147a26601-maas-api-xks-ca" pod "4de4a3ad-a331-4ef2-b1ec-e75147a26601" (UID: "4de4a3ad-a331-4ef2-b1ec-e75147a26601"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:09.833939 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:09.833920 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4de4a3ad-a331-4ef2-b1ec-e75147a26601-kube-api-access-bsjkb" pod "4de4a3ad-a331-4ef2-b1ec-e75147a26601" (UID: "4de4a3ad-a331-4ef2-b1ec-e75147a26601"). InnerVolumeSpecName "kube-api-access-bsjkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:09.933293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:09.933237 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsjkb\" (UniqueName: \"kubernetes.io/projected/4de4a3ad-a331-4ef2-b1ec-e75147a26601-kube-api-access-bsjkb\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:09.933293 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:09.933275 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4de4a3ad-a331-4ef2-b1ec-e75147a26601-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:10.533330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:10.533297 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz" Oct 30 21:58:10.533330 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:10.533324 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz" event={"ID":"4de4a3ad-a331-4ef2-b1ec-e75147a26601","Type":"ContainerDied","Data":"692788d51c12d55e43553dae1b98ba191080d1c5a619f9153da20668d9b1bc02"} Oct 30 21:58:10.533597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:10.533372 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="692788d51c12d55e43553dae1b98ba191080d1c5a619f9153da20668d9b1bc02" Oct 30 21:58:11.434130 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.434048 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:58:11.521248 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.521208 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44"] Oct 30 21:58:11.526998 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.526966 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-43c070-h4m44"] Oct 30 21:58:11.542526 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.542484 4926 generic.go:356] "Generic (PLEG): container finished" podID="73a624ed-4a10-48c0-9a7a-002ebb6114df" containerID="7f92220df6c2c9884681329f95c7a94de88e08fda0c2be135bca9985d8cdfab5" exitCode=0 Oct 30 21:58:11.542691 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.542551 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" Oct 30 21:58:11.542691 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.542565 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" event={"ID":"73a624ed-4a10-48c0-9a7a-002ebb6114df","Type":"ContainerDied","Data":"7f92220df6c2c9884681329f95c7a94de88e08fda0c2be135bca9985d8cdfab5"} Oct 30 21:58:11.542691 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.542633 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg" event={"ID":"73a624ed-4a10-48c0-9a7a-002ebb6114df","Type":"ContainerDied","Data":"66bc45417f6d5544fa9ebbed1c0cde02046a09b1f044f17c61f2c3382cc9b244"} Oct 30 21:58:11.542691 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.542659 4926 scope.go:122] "RemoveContainer" containerID="7f92220df6c2c9884681329f95c7a94de88e08fda0c2be135bca9985d8cdfab5" Oct 30 21:58:11.553960 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.553937 4926 scope.go:122] "RemoveContainer" containerID="7f92220df6c2c9884681329f95c7a94de88e08fda0c2be135bca9985d8cdfab5" Oct 30 21:58:11.554278 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:58:11.554258 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f92220df6c2c9884681329f95c7a94de88e08fda0c2be135bca9985d8cdfab5\": container with ID starting with 7f92220df6c2c9884681329f95c7a94de88e08fda0c2be135bca9985d8cdfab5 not found: ID does not exist" containerID="7f92220df6c2c9884681329f95c7a94de88e08fda0c2be135bca9985d8cdfab5" Oct 30 21:58:11.554348 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.554290 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f92220df6c2c9884681329f95c7a94de88e08fda0c2be135bca9985d8cdfab5"} err="failed to get container status \"7f92220df6c2c9884681329f95c7a94de88e08fda0c2be135bca9985d8cdfab5\": rpc error: code = NotFound desc = could not find container \"7f92220df6c2c9884681329f95c7a94de88e08fda0c2be135bca9985d8cdfab5\": container with ID starting with 7f92220df6c2c9884681329f95c7a94de88e08fda0c2be135bca9985d8cdfab5 not found: ID does not exist" Oct 30 21:58:11.555529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.555504 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-podinfo\") pod \"73a624ed-4a10-48c0-9a7a-002ebb6114df\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " Oct 30 21:58:11.555837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.555553 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73a624ed-4a10-48c0-9a7a-002ebb6114df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a624ed-4a10-48c0-9a7a-002ebb6114df-istiod-ca-cert\") pod \"73a624ed-4a10-48c0-9a7a-002ebb6114df\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " Oct 30 21:58:11.555837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.555589 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-credential-socket\") pod \"73a624ed-4a10-48c0-9a7a-002ebb6114df\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " Oct 30 21:58:11.555837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.555652 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-envoy\") pod \"73a624ed-4a10-48c0-9a7a-002ebb6114df\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " Oct 30 21:58:11.555837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.555686 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a624ed-4a10-48c0-9a7a-002ebb6114df-kube-api-access-qv9kw\" (UniqueName: \"kubernetes.io/projected/73a624ed-4a10-48c0-9a7a-002ebb6114df-kube-api-access-qv9kw\") pod \"73a624ed-4a10-48c0-9a7a-002ebb6114df\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " Oct 30 21:58:11.555837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.555760 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-data\") pod \"73a624ed-4a10-48c0-9a7a-002ebb6114df\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " Oct 30 21:58:11.555837 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.555820 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-token\" (UniqueName: \"kubernetes.io/projected/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-token\") pod \"73a624ed-4a10-48c0-9a7a-002ebb6114df\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " Oct 30 21:58:11.556221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.555887 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-workload-socket\") pod \"73a624ed-4a10-48c0-9a7a-002ebb6114df\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " Oct 30 21:58:11.556221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.555965 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-credential-socket" pod "73a624ed-4a10-48c0-9a7a-002ebb6114df" (UID: "73a624ed-4a10-48c0-9a7a-002ebb6114df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:11.556221 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.556142 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-workload-socket" pod "73a624ed-4a10-48c0-9a7a-002ebb6114df" (UID: "73a624ed-4a10-48c0-9a7a-002ebb6114df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:11.556384 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.556215 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-data" pod "73a624ed-4a10-48c0-9a7a-002ebb6114df" (UID: "73a624ed-4a10-48c0-9a7a-002ebb6114df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:11.556492 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.556467 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-workload-certs\") pod \"73a624ed-4a10-48c0-9a7a-002ebb6114df\" (UID: \"73a624ed-4a10-48c0-9a7a-002ebb6114df\") " Oct 30 21:58:11.556756 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.556723 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-workload-certs" pod "73a624ed-4a10-48c0-9a7a-002ebb6114df" (UID: "73a624ed-4a10-48c0-9a7a-002ebb6114df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:11.558218 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.556931 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a624ed-4a10-48c0-9a7a-002ebb6114df-istiod-ca-cert" pod "73a624ed-4a10-48c0-9a7a-002ebb6114df" (UID: "73a624ed-4a10-48c0-9a7a-002ebb6114df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:11.558374 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.556957 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:11.558495 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.558481 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:11.558585 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.558574 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:11.558666 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.558655 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:11.559359 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.559330 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-envoy" pod "73a624ed-4a10-48c0-9a7a-002ebb6114df" (UID: "73a624ed-4a10-48c0-9a7a-002ebb6114df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:11.559512 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.559489 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-token" pod "73a624ed-4a10-48c0-9a7a-002ebb6114df" (UID: "73a624ed-4a10-48c0-9a7a-002ebb6114df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:11.562114 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.562087 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-podinfo" pod "73a624ed-4a10-48c0-9a7a-002ebb6114df" (UID: "73a624ed-4a10-48c0-9a7a-002ebb6114df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:58:11.563020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.562992 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a624ed-4a10-48c0-9a7a-002ebb6114df-kube-api-access-qv9kw" pod "73a624ed-4a10-48c0-9a7a-002ebb6114df" (UID: "73a624ed-4a10-48c0-9a7a-002ebb6114df"). InnerVolumeSpecName "kube-api-access-qv9kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:11.660025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.659937 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:11.660025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.659971 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:11.660025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.659981 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73a624ed-4a10-48c0-9a7a-002ebb6114df-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:11.660025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.659990 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73a624ed-4a10-48c0-9a7a-002ebb6114df-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:11.660025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.659999 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv9kw\" (UniqueName: \"kubernetes.io/projected/73a624ed-4a10-48c0-9a7a-002ebb6114df-kube-api-access-qv9kw\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:11.896377 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.896340 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg"] Oct 30 21:58:11.907525 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:11.907492 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-908469-openshift-default-54f468f759-nxmmg"] Oct 30 21:58:13.017025 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:13.016990 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06ba0c21-248b-4f6b-889a-78ac94c2a07e" path="/var/lib/kubelet/pods/06ba0c21-248b-4f6b-889a-78ac94c2a07e/volumes" Oct 30 21:58:13.017529 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:13.017482 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73a624ed-4a10-48c0-9a7a-002ebb6114df" path="/var/lib/kubelet/pods/73a624ed-4a10-48c0-9a7a-002ebb6114df/volumes" Oct 30 21:58:18.580703 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:18.580658 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz"] Oct 30 21:58:18.589930 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:18.589895 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95692f-t89qz"] Oct 30 21:58:19.026209 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:19.026164 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4de4a3ad-a331-4ef2-b1ec-e75147a26601" path="/var/lib/kubelet/pods/4de4a3ad-a331-4ef2-b1ec-e75147a26601/volumes" Oct 30 21:58:19.340723 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:19.340632 4926 scope.go:122] "RemoveContainer" containerID="2e1b19aeaf8c889183fd369944cccfcf020c57eee2ff3668dfaa53696102b49c" Oct 30 21:58:19.351593 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:19.351566 4926 scope.go:122] "RemoveContainer" containerID="1d3921eb970d966cb7d5b7788ca9ffc9ca469d26f7ee4d1906e2a3c7bd9b17f1" Oct 30 21:58:19.363522 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:19.363491 4926 scope.go:122] "RemoveContainer" containerID="d88dec00f9a0ce7ee358e97497567712ea5016439789de87f5b7974128cd8b92" Oct 30 21:58:23.411313 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:23.411266 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:58:25.031481 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:25.031439 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4016884f-c6vqr"] Oct 30 21:58:25.042088 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:25.042055 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4016884f-c6vqr"] Oct 30 21:58:25.775271 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:25.775232 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm"] Oct 30 21:58:25.775523 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:25.775468 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" podUID="239b39fd-6aa2-4f03-8aef-d46a83aafc0c" containerName="istio-proxy" containerID="cri-o://ff11bf4d6d3046adeabfda897c781fe65d4bf5f963245c6188599df871b7c6b0" gracePeriod=30 Oct 30 21:58:27.016592 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:27.016548 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f40e13d6-fe4d-4ff3-b48d-4e57d1985db8" path="/var/lib/kubelet/pods/f40e13d6-fe4d-4ff3-b48d-4e57d1985db8/volumes" Oct 30 21:58:31.047345 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.047273 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:58:31.165974 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.165931 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istiod-ca-cert\") pod \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " Oct 30 21:58:31.166203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.166097 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-token\" (UniqueName: \"kubernetes.io/projected/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-token\") pod \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " Oct 30 21:58:31.166203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.166147 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-podinfo\") pod \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " Oct 30 21:58:31.166203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.166181 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-envoy\") pod \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " Oct 30 21:58:31.166389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.166216 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-workload-certs\") pod \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " Oct 30 21:58:31.166389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.166247 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-kube-api-access-fzcdf\" (UniqueName: \"kubernetes.io/projected/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-kube-api-access-fzcdf\") pod \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " Oct 30 21:58:31.166389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.166317 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-credential-socket\") pod \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " Oct 30 21:58:31.166389 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.166360 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-workload-socket\") pod \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " Oct 30 21:58:31.166606 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.166390 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-data\") pod \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\" (UID: \"239b39fd-6aa2-4f03-8aef-d46a83aafc0c\") " Oct 30 21:58:31.166606 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.166407 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istiod-ca-cert" pod "239b39fd-6aa2-4f03-8aef-d46a83aafc0c" (UID: "239b39fd-6aa2-4f03-8aef-d46a83aafc0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:31.166728 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.166708 4926 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istiod-ca-cert\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:31.167224 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.166802 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-workload-socket" pod "239b39fd-6aa2-4f03-8aef-d46a83aafc0c" (UID: "239b39fd-6aa2-4f03-8aef-d46a83aafc0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:31.167224 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.166943 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-data" pod "239b39fd-6aa2-4f03-8aef-d46a83aafc0c" (UID: "239b39fd-6aa2-4f03-8aef-d46a83aafc0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:31.167224 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.167096 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-workload-certs" pod "239b39fd-6aa2-4f03-8aef-d46a83aafc0c" (UID: "239b39fd-6aa2-4f03-8aef-d46a83aafc0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:31.167224 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.167159 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-credential-socket" pod "239b39fd-6aa2-4f03-8aef-d46a83aafc0c" (UID: "239b39fd-6aa2-4f03-8aef-d46a83aafc0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:31.168973 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.168942 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-kube-api-access-fzcdf" pod "239b39fd-6aa2-4f03-8aef-d46a83aafc0c" (UID: "239b39fd-6aa2-4f03-8aef-d46a83aafc0c"). InnerVolumeSpecName "kube-api-access-fzcdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:31.169132 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.168989 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-token" pod "239b39fd-6aa2-4f03-8aef-d46a83aafc0c" (UID: "239b39fd-6aa2-4f03-8aef-d46a83aafc0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:31.169132 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.169095 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-podinfo" pod "239b39fd-6aa2-4f03-8aef-d46a83aafc0c" (UID: "239b39fd-6aa2-4f03-8aef-d46a83aafc0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:58:31.169132 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.169106 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-envoy" pod "239b39fd-6aa2-4f03-8aef-d46a83aafc0c" (UID: "239b39fd-6aa2-4f03-8aef-d46a83aafc0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:31.268381 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.268331 4926 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-podinfo\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:31.268381 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.268377 4926 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-envoy\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:31.268381 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.268392 4926 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-workload-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:31.268725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.268407 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzcdf\" (UniqueName: \"kubernetes.io/projected/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-kube-api-access-fzcdf\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:31.268725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.268421 4926 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-credential-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:31.268725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.268435 4926 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-workload-socket\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:31.268725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.268448 4926 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-data\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:31.268725 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.268461 4926 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/239b39fd-6aa2-4f03-8aef-d46a83aafc0c-istio-token\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:31.269246 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.269215 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf"] Oct 30 21:58:31.276996 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.276969 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb"] Oct 30 21:58:31.293372 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.293335 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx"] Oct 30 21:58:31.300177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.300109 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc"] Oct 30 21:58:31.717008 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.716973 4926 generic.go:356] "Generic (PLEG): container finished" podID="239b39fd-6aa2-4f03-8aef-d46a83aafc0c" containerID="ff11bf4d6d3046adeabfda897c781fe65d4bf5f963245c6188599df871b7c6b0" exitCode=0 Oct 30 21:58:31.717203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.717067 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" event={"ID":"239b39fd-6aa2-4f03-8aef-d46a83aafc0c","Type":"ContainerDied","Data":"ff11bf4d6d3046adeabfda897c781fe65d4bf5f963245c6188599df871b7c6b0"} Oct 30 21:58:31.717203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.717080 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" Oct 30 21:58:31.717203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.717153 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm" event={"ID":"239b39fd-6aa2-4f03-8aef-d46a83aafc0c","Type":"ContainerDied","Data":"9f6466b61259abd697caaec350a24ca2f7c35ccf0dc2cef9f14c56a33bd63478"} Oct 30 21:58:31.717203 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.717177 4926 scope.go:122] "RemoveContainer" containerID="ff11bf4d6d3046adeabfda897c781fe65d4bf5f963245c6188599df871b7c6b0" Oct 30 21:58:31.729370 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.729352 4926 scope.go:122] "RemoveContainer" containerID="ff11bf4d6d3046adeabfda897c781fe65d4bf5f963245c6188599df871b7c6b0" Oct 30 21:58:31.729709 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:58:31.729683 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff11bf4d6d3046adeabfda897c781fe65d4bf5f963245c6188599df871b7c6b0\": container with ID starting with ff11bf4d6d3046adeabfda897c781fe65d4bf5f963245c6188599df871b7c6b0 not found: ID does not exist" containerID="ff11bf4d6d3046adeabfda897c781fe65d4bf5f963245c6188599df871b7c6b0" Oct 30 21:58:31.729813 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.729721 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff11bf4d6d3046adeabfda897c781fe65d4bf5f963245c6188599df871b7c6b0"} err="failed to get container status \"ff11bf4d6d3046adeabfda897c781fe65d4bf5f963245c6188599df871b7c6b0\": rpc error: code = NotFound desc = could not find container \"ff11bf4d6d3046adeabfda897c781fe65d4bf5f963245c6188599df871b7c6b0\": container with ID starting with ff11bf4d6d3046adeabfda897c781fe65d4bf5f963245c6188599df871b7c6b0 not found: ID does not exist" Oct 30 21:58:31.766098 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.766071 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm"] Oct 30 21:58:31.777343 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:31.777315 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95692f-openshift-default-59b6f8c66c-gtphm"] Oct 30 21:58:33.015362 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:33.015325 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="239b39fd-6aa2-4f03-8aef-d46a83aafc0c" path="/var/lib/kubelet/pods/239b39fd-6aa2-4f03-8aef-d46a83aafc0c/volumes" Oct 30 21:58:33.027158 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:33.027125 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46"] Oct 30 21:58:33.037858 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:33.037830 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3553cc4-ncx46"] Oct 30 21:58:35.014981 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:35.014945 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b09edca-4699-4a05-9109-d8b0b11af8bf" path="/var/lib/kubelet/pods/1b09edca-4699-4a05-9109-d8b0b11af8bf/volumes" Oct 30 21:58:37.484768 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:58:37.484729 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:58:37.492382 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.492352 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l"] Oct 30 21:58:37.492777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.492756 4926 cpu_manager.go:414] "containerMap: removing container" podUID="239b39fd-6aa2-4f03-8aef-d46a83aafc0c" containerName="istio-proxy" Oct 30 21:58:37.492777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.492771 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="239b39fd-6aa2-4f03-8aef-d46a83aafc0c" containerName="istio-proxy" Oct 30 21:58:37.492888 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.492826 4926 cpu_manager.go:414] "containerMap: removing container" podUID="4de4a3ad-a331-4ef2-b1ec-e75147a26601" containerName="revoke-keys" Oct 30 21:58:37.492888 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.492836 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4de4a3ad-a331-4ef2-b1ec-e75147a26601" containerName="revoke-keys" Oct 30 21:58:37.492888 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.492857 4926 cpu_manager.go:414] "containerMap: removing container" podUID="73a624ed-4a10-48c0-9a7a-002ebb6114df" containerName="istio-proxy" Oct 30 21:58:37.492888 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.492865 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a624ed-4a10-48c0-9a7a-002ebb6114df" containerName="istio-proxy" Oct 30 21:58:37.493068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.492965 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="4de4a3ad-a331-4ef2-b1ec-e75147a26601" containerName="revoke-keys" Oct 30 21:58:37.493068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.492979 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="73a624ed-4a10-48c0-9a7a-002ebb6114df" containerName="istio-proxy" Oct 30 21:58:37.493068 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.492990 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="239b39fd-6aa2-4f03-8aef-d46a83aafc0c" containerName="istio-proxy" Oct 30 21:58:37.497777 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.497754 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l" Oct 30 21:58:37.525431 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.525387 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l"] Oct 30 21:58:37.592618 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.592587 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 21:58:37.592618 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.592611 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:58:37.592847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.592600 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:58:37.592847 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.592586 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 21:58:37.631978 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.631939 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5285d5dc-80ad-40c6-b749-9c215955a4bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bf396ba-5fx8l\" (UID: \"5285d5dc-80ad-40c6-b749-9c215955a4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l" Oct 30 21:58:37.632198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.631995 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c77hx\" (UniqueName: \"kubernetes.io/projected/5285d5dc-80ad-40c6-b749-9c215955a4bd-kube-api-access-c77hx\") pod \"maas-api-revoke-sub-2bf396ba-5fx8l\" (UID: \"5285d5dc-80ad-40c6-b749-9c215955a4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l" Oct 30 21:58:37.632198 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.632115 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5285d5dc-80ad-40c6-b749-9c215955a4bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bf396ba-5fx8l\" (UID: \"5285d5dc-80ad-40c6-b749-9c215955a4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l" Oct 30 21:58:37.733457 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.733413 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c77hx\" (UniqueName: \"kubernetes.io/projected/5285d5dc-80ad-40c6-b749-9c215955a4bd-kube-api-access-c77hx\") pod \"maas-api-revoke-sub-2bf396ba-5fx8l\" (UID: \"5285d5dc-80ad-40c6-b749-9c215955a4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l" Oct 30 21:58:37.733650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.733479 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5285d5dc-80ad-40c6-b749-9c215955a4bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bf396ba-5fx8l\" (UID: \"5285d5dc-80ad-40c6-b749-9c215955a4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l" Oct 30 21:58:37.733650 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.733592 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5285d5dc-80ad-40c6-b749-9c215955a4bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bf396ba-5fx8l\" (UID: \"5285d5dc-80ad-40c6-b749-9c215955a4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l" Oct 30 21:58:37.734356 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.734331 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5285d5dc-80ad-40c6-b749-9c215955a4bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bf396ba-5fx8l\" (UID: \"5285d5dc-80ad-40c6-b749-9c215955a4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l" Oct 30 21:58:37.736079 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.736013 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5285d5dc-80ad-40c6-b749-9c215955a4bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bf396ba-5fx8l\" (UID: \"5285d5dc-80ad-40c6-b749-9c215955a4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l" Oct 30 21:58:37.751195 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.751168 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c77hx\" (UniqueName: \"kubernetes.io/projected/5285d5dc-80ad-40c6-b749-9c215955a4bd-kube-api-access-c77hx\") pod \"maas-api-revoke-sub-2bf396ba-5fx8l\" (UID: \"5285d5dc-80ad-40c6-b749-9c215955a4bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l" Oct 30 21:58:37.834126 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:37.834091 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l" Oct 30 21:58:38.158287 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:38.158257 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l"] Oct 30 21:58:38.160062 ip-10-0-142-3 kubenswrapper[4926]: W1030 21:58:38.160005 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5285d5dc_80ad_40c6_b749_9c215955a4bd.slice/crio-54db388f8ccaa8d52c0ea1cff358b3fadc789a8e90dc5bee7331189b69552cea WatchSource:0}: Error finding container 54db388f8ccaa8d52c0ea1cff358b3fadc789a8e90dc5bee7331189b69552cea: Status 404 returned error can't find the container with id 54db388f8ccaa8d52c0ea1cff358b3fadc789a8e90dc5bee7331189b69552cea Oct 30 21:58:38.774950 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:38.774850 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l" event={"ID":"5285d5dc-80ad-40c6-b749-9c215955a4bd","Type":"ContainerStarted","Data":"91ca1091e2e8a2715164e9d04bf02286a3689f0f1ce6abd17d6774f84b87717a"} Oct 30 21:58:38.774950 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:38.774907 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l" event={"ID":"5285d5dc-80ad-40c6-b749-9c215955a4bd","Type":"ContainerStarted","Data":"54db388f8ccaa8d52c0ea1cff358b3fadc789a8e90dc5bee7331189b69552cea"} Oct 30 21:58:38.815620 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:38.815557 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l" podStartSLOduration=1.815543342 podStartE2EDuration="1.815543342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:38.813050128 +0000 UTC m=+3505.934123880" watchObservedRunningTime="2026-10-30 21:58:38.815543342 +0000 UTC m=+3505.936617088" Oct 30 21:58:39.783792 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:39.783749 4926 generic.go:356] "Generic (PLEG): container finished" podID="5285d5dc-80ad-40c6-b749-9c215955a4bd" containerID="91ca1091e2e8a2715164e9d04bf02286a3689f0f1ce6abd17d6774f84b87717a" exitCode=0 Oct 30 21:58:39.784292 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:39.783839 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l" event={"ID":"5285d5dc-80ad-40c6-b749-9c215955a4bd","Type":"ContainerDied","Data":"91ca1091e2e8a2715164e9d04bf02286a3689f0f1ce6abd17d6774f84b87717a"} Oct 30 21:58:40.934081 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:40.934054 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l" Oct 30 21:58:41.069211 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:41.069120 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5285d5dc-80ad-40c6-b749-9c215955a4bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5285d5dc-80ad-40c6-b749-9c215955a4bd-maas-api-xks-ca\") pod \"5285d5dc-80ad-40c6-b749-9c215955a4bd\" (UID: \"5285d5dc-80ad-40c6-b749-9c215955a4bd\") " Oct 30 21:58:41.069211 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:41.069197 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5285d5dc-80ad-40c6-b749-9c215955a4bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5285d5dc-80ad-40c6-b749-9c215955a4bd-maas-api-openshift-ca\") pod \"5285d5dc-80ad-40c6-b749-9c215955a4bd\" (UID: \"5285d5dc-80ad-40c6-b749-9c215955a4bd\") " Oct 30 21:58:41.069452 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:41.069266 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5285d5dc-80ad-40c6-b749-9c215955a4bd-kube-api-access-c77hx\" (UniqueName: \"kubernetes.io/projected/5285d5dc-80ad-40c6-b749-9c215955a4bd-kube-api-access-c77hx\") pod \"5285d5dc-80ad-40c6-b749-9c215955a4bd\" (UID: \"5285d5dc-80ad-40c6-b749-9c215955a4bd\") " Oct 30 21:58:41.069685 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:41.069661 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5285d5dc-80ad-40c6-b749-9c215955a4bd-maas-api-openshift-ca" pod "5285d5dc-80ad-40c6-b749-9c215955a4bd" (UID: "5285d5dc-80ad-40c6-b749-9c215955a4bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:58:41.071356 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:41.071331 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5285d5dc-80ad-40c6-b749-9c215955a4bd-maas-api-xks-ca" pod "5285d5dc-80ad-40c6-b749-9c215955a4bd" (UID: "5285d5dc-80ad-40c6-b749-9c215955a4bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:41.071455 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:41.071381 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5285d5dc-80ad-40c6-b749-9c215955a4bd-kube-api-access-c77hx" pod "5285d5dc-80ad-40c6-b749-9c215955a4bd" (UID: "5285d5dc-80ad-40c6-b749-9c215955a4bd"). InnerVolumeSpecName "kube-api-access-c77hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:41.170375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:41.170339 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c77hx\" (UniqueName: \"kubernetes.io/projected/5285d5dc-80ad-40c6-b749-9c215955a4bd-kube-api-access-c77hx\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:41.170375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:41.170372 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5285d5dc-80ad-40c6-b749-9c215955a4bd-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:41.170375 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:41.170383 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5285d5dc-80ad-40c6-b749-9c215955a4bd-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:41.800390 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:41.800349 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l" Oct 30 21:58:41.800390 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:41.800378 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l" event={"ID":"5285d5dc-80ad-40c6-b749-9c215955a4bd","Type":"ContainerDied","Data":"54db388f8ccaa8d52c0ea1cff358b3fadc789a8e90dc5bee7331189b69552cea"} Oct 30 21:58:41.800687 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:41.800429 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54db388f8ccaa8d52c0ea1cff358b3fadc789a8e90dc5bee7331189b69552cea" Oct 30 21:58:43.026505 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:43.026461 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz"] Oct 30 21:58:43.037539 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:43.037507 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0472f399-drpbz"] Oct 30 21:58:45.016928 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:45.016888 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0c5e81a-a031-4691-965f-1065db931bc4" path="/var/lib/kubelet/pods/b0c5e81a-a031-4691-965f-1065db931bc4/volumes" Oct 30 21:58:46.298286 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.298175 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" containerID="cri-o://8c0468af1520fe80e8b425f4525176680c01c0fbab25e68cdaef2e64f02c681a" gracePeriod=45 Oct 30 21:58:46.300108 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.300051 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" containerID="cri-o://eb680a58d7702a24a9211fd9c44d41083d67581d1a06a11f418a084f92f00acf" gracePeriod=45 Oct 30 21:58:46.316673 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.316607 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" containerID="cri-o://e035d50d5ae960a3ed6d8881755e550630c1ac4feb05d2fe90bd99c989eeab9a" gracePeriod=45 Oct 30 21:58:46.321778 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.321715 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" containerID="cri-o://4870892cdb213c59c1442b76b7c1c3d45a1f2921f7c88f5c8205826ce573240e" gracePeriod=45 Oct 30 21:58:46.367662 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.367616 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": dial tcp 10.132.0.74:8000: connect: connection refused" Oct 30 21:58:46.518537 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.518491 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": dial tcp 10.132.0.60:8000: connect: connection refused" Oct 30 21:58:46.605671 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.604224 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:58:46.622443 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.622415 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:58:46.642211 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.642182 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:58:46.666601 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.666575 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:58:46.720922 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.720856 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-home\" (UniqueName: \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-home\") pod \"1cc79717-77ce-4405-b102-ff904f2b345f\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " Oct 30 21:58:46.720922 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.720918 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-tmp-dir\") pod \"2ae4a221-e691-49b2-bafa-35e50ea1bead\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " Oct 30 21:58:46.721597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.720973 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-dshm\") pod \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " Oct 30 21:58:46.721597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.721009 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-kube-api-access-hhx4h\" (UniqueName: \"kubernetes.io/projected/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-kube-api-access-hhx4h\") pod \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " Oct 30 21:58:46.721597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.721059 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae4a221-e691-49b2-bafa-35e50ea1bead-kube-api-access-zdgfx\" (UniqueName: \"kubernetes.io/projected/2ae4a221-e691-49b2-bafa-35e50ea1bead-kube-api-access-zdgfx\") pod \"2ae4a221-e691-49b2-bafa-35e50ea1bead\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " Oct 30 21:58:46.721597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.721132 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ae4a221-e691-49b2-bafa-35e50ea1bead-tls-certs\" (UniqueName: \"kubernetes.io/secret/2ae4a221-e691-49b2-bafa-35e50ea1bead-tls-certs\") pod \"2ae4a221-e691-49b2-bafa-35e50ea1bead\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " Oct 30 21:58:46.721597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.721158 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-model-cache\") pod \"1cc79717-77ce-4405-b102-ff904f2b345f\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " Oct 30 21:58:46.721597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.721193 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cc79717-77ce-4405-b102-ff904f2b345f-tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc79717-77ce-4405-b102-ff904f2b345f-tls-certs\") pod \"1cc79717-77ce-4405-b102-ff904f2b345f\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " Oct 30 21:58:46.721597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.721225 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-tmp-dir\") pod \"1cc79717-77ce-4405-b102-ff904f2b345f\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " Oct 30 21:58:46.721597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.721254 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-home\" (UniqueName: \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-home\") pod \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " Oct 30 21:58:46.721597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.721280 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-tls-certs\") pod \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " Oct 30 21:58:46.721597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.721322 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cc79717-77ce-4405-b102-ff904f2b345f-kube-api-access-j59bv\" (UniqueName: \"kubernetes.io/projected/1cc79717-77ce-4405-b102-ff904f2b345f-kube-api-access-j59bv\") pod \"1cc79717-77ce-4405-b102-ff904f2b345f\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " Oct 30 21:58:46.721597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.721362 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-model-cache\") pod \"2ae4a221-e691-49b2-bafa-35e50ea1bead\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " Oct 30 21:58:46.721597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.721400 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-tmp-dir\") pod \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " Oct 30 21:58:46.721597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.721437 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-home\" (UniqueName: \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-home\") pod \"2ae4a221-e691-49b2-bafa-35e50ea1bead\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " Oct 30 21:58:46.721597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.721459 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-dshm\") pod \"2ae4a221-e691-49b2-bafa-35e50ea1bead\" (UID: \"2ae4a221-e691-49b2-bafa-35e50ea1bead\") " Oct 30 21:58:46.721597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.721490 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-dshm\") pod \"1cc79717-77ce-4405-b102-ff904f2b345f\" (UID: \"1cc79717-77ce-4405-b102-ff904f2b345f\") " Oct 30 21:58:46.721597 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.721534 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-model-cache\") pod \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\" (UID: \"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d\") " Oct 30 21:58:46.722522 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.721637 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-home" pod "1cc79717-77ce-4405-b102-ff904f2b345f" (UID: "1cc79717-77ce-4405-b102-ff904f2b345f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:46.722522 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.721888 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-model-cache" pod "d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" (UID: "d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:46.722522 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.722104 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.722522 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.722108 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-model-cache" pod "1cc79717-77ce-4405-b102-ff904f2b345f" (UID: "1cc79717-77ce-4405-b102-ff904f2b345f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:46.722522 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.722119 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.722522 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.722321 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-model-cache" pod "2ae4a221-e691-49b2-bafa-35e50ea1bead" (UID: "2ae4a221-e691-49b2-bafa-35e50ea1bead"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:46.722522 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.722331 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-tmp-dir" pod "1cc79717-77ce-4405-b102-ff904f2b345f" (UID: "1cc79717-77ce-4405-b102-ff904f2b345f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:46.722522 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.722514 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-home" pod "d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" (UID: "d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:46.723059 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.722883 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-tmp-dir" pod "d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" (UID: "d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:46.723154 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.723129 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-home" pod "2ae4a221-e691-49b2-bafa-35e50ea1bead" (UID: "2ae4a221-e691-49b2-bafa-35e50ea1bead"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:46.723710 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.723681 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-tmp-dir" pod "2ae4a221-e691-49b2-bafa-35e50ea1bead" (UID: "2ae4a221-e691-49b2-bafa-35e50ea1bead"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:46.725875 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.725852 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-dshm" pod "d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" (UID: "d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:46.726003 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.725981 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae4a221-e691-49b2-bafa-35e50ea1bead-tls-certs" pod "2ae4a221-e691-49b2-bafa-35e50ea1bead" (UID: "2ae4a221-e691-49b2-bafa-35e50ea1bead"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:46.726242 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.726216 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cc79717-77ce-4405-b102-ff904f2b345f-tls-certs" pod "1cc79717-77ce-4405-b102-ff904f2b345f" (UID: "1cc79717-77ce-4405-b102-ff904f2b345f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:46.726413 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.726389 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-dshm" pod "1cc79717-77ce-4405-b102-ff904f2b345f" (UID: "1cc79717-77ce-4405-b102-ff904f2b345f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:46.727530 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.727122 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-tls-certs" pod "d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" (UID: "d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:46.727638 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.727552 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-kube-api-access-hhx4h" pod "d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" (UID: "d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d"). InnerVolumeSpecName "kube-api-access-hhx4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:46.727699 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.727640 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae4a221-e691-49b2-bafa-35e50ea1bead-kube-api-access-zdgfx" pod "2ae4a221-e691-49b2-bafa-35e50ea1bead" (UID: "2ae4a221-e691-49b2-bafa-35e50ea1bead"). InnerVolumeSpecName "kube-api-access-zdgfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:46.727741 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.727689 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-dshm" pod "2ae4a221-e691-49b2-bafa-35e50ea1bead" (UID: "2ae4a221-e691-49b2-bafa-35e50ea1bead"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:46.727943 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.727918 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cc79717-77ce-4405-b102-ff904f2b345f-kube-api-access-j59bv" pod "1cc79717-77ce-4405-b102-ff904f2b345f" (UID: "1cc79717-77ce-4405-b102-ff904f2b345f"). InnerVolumeSpecName "kube-api-access-j59bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:46.823347 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823265 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-model-cache\") pod \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " Oct 30 21:58:46.823530 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823357 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-dshm\") pod \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " Oct 30 21:58:46.823530 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823433 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-kube-api-access-cmk66\" (UniqueName: \"kubernetes.io/projected/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-kube-api-access-cmk66\") pod \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " Oct 30 21:58:46.823530 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823476 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-home\" (UniqueName: \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-home\") pod \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " Oct 30 21:58:46.823530 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823512 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-tmp-dir\") pod \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " Oct 30 21:58:46.823781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823558 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-tls-certs\" (UniqueName: \"kubernetes.io/secret/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-tls-certs\") pod \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\" (UID: \"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4\") " Oct 30 21:58:46.823781 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823610 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-model-cache" pod "d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" (UID: "d0cb8157-aaea-41e8-af1a-aef07b8a9bc4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:46.823892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823788 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhx4h\" (UniqueName: \"kubernetes.io/projected/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-kube-api-access-hhx4h\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.823892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823808 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdgfx\" (UniqueName: \"kubernetes.io/projected/2ae4a221-e691-49b2-bafa-35e50ea1bead-kube-api-access-zdgfx\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.823892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823823 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2ae4a221-e691-49b2-bafa-35e50ea1bead-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.823892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823825 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-tmp-dir" pod "d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" (UID: "d0cb8157-aaea-41e8-af1a-aef07b8a9bc4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:46.823892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823838 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.823892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823852 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1cc79717-77ce-4405-b102-ff904f2b345f-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.823892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823864 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.823892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823878 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.823892 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823891 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.824444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823904 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j59bv\" (UniqueName: \"kubernetes.io/projected/1cc79717-77ce-4405-b102-ff904f2b345f-kube-api-access-j59bv\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.824444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823917 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.824444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823931 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.824444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823943 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.824444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823955 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.824444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823966 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.824444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823977 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1cc79717-77ce-4405-b102-ff904f2b345f-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.824444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823988 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2ae4a221-e691-49b2-bafa-35e50ea1bead-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.824444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.823999 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.824444 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.824058 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-home" pod "d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" (UID: "d0cb8157-aaea-41e8-af1a-aef07b8a9bc4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:46.825925 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.825901 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-tls-certs" pod "d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" (UID: "d0cb8157-aaea-41e8-af1a-aef07b8a9bc4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:58:46.826071 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.826049 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-kube-api-access-cmk66" pod "d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" (UID: "d0cb8157-aaea-41e8-af1a-aef07b8a9bc4"). InnerVolumeSpecName "kube-api-access-cmk66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:58:46.826136 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.826081 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-dshm" pod "d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" (UID: "d0cb8157-aaea-41e8-af1a-aef07b8a9bc4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:58:46.840357 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.840329 4926 generic.go:356] "Generic (PLEG): container finished" podID="1cc79717-77ce-4405-b102-ff904f2b345f" containerID="4870892cdb213c59c1442b76b7c1c3d45a1f2921f7c88f5c8205826ce573240e" exitCode=0 Oct 30 21:58:46.840470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.840370 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" Oct 30 21:58:46.840470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.840412 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" event={"ID":"1cc79717-77ce-4405-b102-ff904f2b345f","Type":"ContainerDied","Data":"4870892cdb213c59c1442b76b7c1c3d45a1f2921f7c88f5c8205826ce573240e"} Oct 30 21:58:46.840470 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.840465 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc" event={"ID":"1cc79717-77ce-4405-b102-ff904f2b345f","Type":"ContainerDied","Data":"f64ece6ba5c991549d413be4908e19bc226654b01b0c7fc4f58f24841ec93513"} Oct 30 21:58:46.840678 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.840489 4926 scope.go:122] "RemoveContainer" containerID="4870892cdb213c59c1442b76b7c1c3d45a1f2921f7c88f5c8205826ce573240e" Oct 30 21:58:46.842177 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.842155 4926 generic.go:356] "Generic (PLEG): container finished" podID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerID="8c0468af1520fe80e8b425f4525176680c01c0fbab25e68cdaef2e64f02c681a" exitCode=0 Oct 30 21:58:46.842301 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.842268 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" event={"ID":"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d","Type":"ContainerDied","Data":"8c0468af1520fe80e8b425f4525176680c01c0fbab25e68cdaef2e64f02c681a"} Oct 30 21:58:46.842301 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.842296 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" event={"ID":"d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d","Type":"ContainerDied","Data":"07ebc80fb63fd3540a9d144624f0a29d21f120266876dca3701f90125cbe2c6c"} Oct 30 21:58:46.842436 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.842357 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf" Oct 30 21:58:46.844823 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.844780 4926 generic.go:356] "Generic (PLEG): container finished" podID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerID="eb680a58d7702a24a9211fd9c44d41083d67581d1a06a11f418a084f92f00acf" exitCode=0 Oct 30 21:58:46.844968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.844826 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" Oct 30 21:58:46.844968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.844894 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" event={"ID":"2ae4a221-e691-49b2-bafa-35e50ea1bead","Type":"ContainerDied","Data":"eb680a58d7702a24a9211fd9c44d41083d67581d1a06a11f418a084f92f00acf"} Oct 30 21:58:46.844968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.844927 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb" event={"ID":"2ae4a221-e691-49b2-bafa-35e50ea1bead","Type":"ContainerDied","Data":"00cf3d05415e37da2476d12c8e3e6db69eafd2277f5c4ca5c8a8e32c47e41b5b"} Oct 30 21:58:46.847622 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.847599 4926 generic.go:356] "Generic (PLEG): container finished" podID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerID="e035d50d5ae960a3ed6d8881755e550630c1ac4feb05d2fe90bd99c989eeab9a" exitCode=0 Oct 30 21:58:46.847726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.847642 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" event={"ID":"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4","Type":"ContainerDied","Data":"e035d50d5ae960a3ed6d8881755e550630c1ac4feb05d2fe90bd99c989eeab9a"} Oct 30 21:58:46.847726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.847662 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" event={"ID":"d0cb8157-aaea-41e8-af1a-aef07b8a9bc4","Type":"ContainerDied","Data":"54fa72b8f73aae25edf1568649be580a142d56580d6a6b5846b3bc237781fec4"} Oct 30 21:58:46.847726 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.847702 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx" Oct 30 21:58:46.853593 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.853572 4926 scope.go:122] "RemoveContainer" containerID="4870892cdb213c59c1442b76b7c1c3d45a1f2921f7c88f5c8205826ce573240e" Oct 30 21:58:46.853888 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:58:46.853865 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4870892cdb213c59c1442b76b7c1c3d45a1f2921f7c88f5c8205826ce573240e\": container with ID starting with 4870892cdb213c59c1442b76b7c1c3d45a1f2921f7c88f5c8205826ce573240e not found: ID does not exist" containerID="4870892cdb213c59c1442b76b7c1c3d45a1f2921f7c88f5c8205826ce573240e" Oct 30 21:58:46.853966 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.853902 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4870892cdb213c59c1442b76b7c1c3d45a1f2921f7c88f5c8205826ce573240e"} err="failed to get container status \"4870892cdb213c59c1442b76b7c1c3d45a1f2921f7c88f5c8205826ce573240e\": rpc error: code = NotFound desc = could not find container \"4870892cdb213c59c1442b76b7c1c3d45a1f2921f7c88f5c8205826ce573240e\": container with ID starting with 4870892cdb213c59c1442b76b7c1c3d45a1f2921f7c88f5c8205826ce573240e not found: ID does not exist" Oct 30 21:58:46.853966 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.853927 4926 scope.go:122] "RemoveContainer" containerID="8c0468af1520fe80e8b425f4525176680c01c0fbab25e68cdaef2e64f02c681a" Oct 30 21:58:46.865110 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.865084 4926 scope.go:122] "RemoveContainer" containerID="8c0468af1520fe80e8b425f4525176680c01c0fbab25e68cdaef2e64f02c681a" Oct 30 21:58:46.865420 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:58:46.865401 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c0468af1520fe80e8b425f4525176680c01c0fbab25e68cdaef2e64f02c681a\": container with ID starting with 8c0468af1520fe80e8b425f4525176680c01c0fbab25e68cdaef2e64f02c681a not found: ID does not exist" containerID="8c0468af1520fe80e8b425f4525176680c01c0fbab25e68cdaef2e64f02c681a" Oct 30 21:58:46.865468 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.865433 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c0468af1520fe80e8b425f4525176680c01c0fbab25e68cdaef2e64f02c681a"} err="failed to get container status \"8c0468af1520fe80e8b425f4525176680c01c0fbab25e68cdaef2e64f02c681a\": rpc error: code = NotFound desc = could not find container \"8c0468af1520fe80e8b425f4525176680c01c0fbab25e68cdaef2e64f02c681a\": container with ID starting with 8c0468af1520fe80e8b425f4525176680c01c0fbab25e68cdaef2e64f02c681a not found: ID does not exist" Oct 30 21:58:46.865468 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.865452 4926 scope.go:122] "RemoveContainer" containerID="eb680a58d7702a24a9211fd9c44d41083d67581d1a06a11f418a084f92f00acf" Oct 30 21:58:46.876968 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.876947 4926 scope.go:122] "RemoveContainer" containerID="eb680a58d7702a24a9211fd9c44d41083d67581d1a06a11f418a084f92f00acf" Oct 30 21:58:46.877232 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:58:46.877210 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb680a58d7702a24a9211fd9c44d41083d67581d1a06a11f418a084f92f00acf\": container with ID starting with eb680a58d7702a24a9211fd9c44d41083d67581d1a06a11f418a084f92f00acf not found: ID does not exist" containerID="eb680a58d7702a24a9211fd9c44d41083d67581d1a06a11f418a084f92f00acf" Oct 30 21:58:46.877364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.877254 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb680a58d7702a24a9211fd9c44d41083d67581d1a06a11f418a084f92f00acf"} err="failed to get container status \"eb680a58d7702a24a9211fd9c44d41083d67581d1a06a11f418a084f92f00acf\": rpc error: code = NotFound desc = could not find container \"eb680a58d7702a24a9211fd9c44d41083d67581d1a06a11f418a084f92f00acf\": container with ID starting with eb680a58d7702a24a9211fd9c44d41083d67581d1a06a11f418a084f92f00acf not found: ID does not exist" Oct 30 21:58:46.877364 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.877277 4926 scope.go:122] "RemoveContainer" containerID="e035d50d5ae960a3ed6d8881755e550630c1ac4feb05d2fe90bd99c989eeab9a" Oct 30 21:58:46.888329 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.888305 4926 scope.go:122] "RemoveContainer" containerID="e035d50d5ae960a3ed6d8881755e550630c1ac4feb05d2fe90bd99c989eeab9a" Oct 30 21:58:46.888581 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:58:46.888558 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e035d50d5ae960a3ed6d8881755e550630c1ac4feb05d2fe90bd99c989eeab9a\": container with ID starting with e035d50d5ae960a3ed6d8881755e550630c1ac4feb05d2fe90bd99c989eeab9a not found: ID does not exist" containerID="e035d50d5ae960a3ed6d8881755e550630c1ac4feb05d2fe90bd99c989eeab9a" Oct 30 21:58:46.888659 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.888594 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e035d50d5ae960a3ed6d8881755e550630c1ac4feb05d2fe90bd99c989eeab9a"} err="failed to get container status \"e035d50d5ae960a3ed6d8881755e550630c1ac4feb05d2fe90bd99c989eeab9a\": rpc error: code = NotFound desc = could not find container \"e035d50d5ae960a3ed6d8881755e550630c1ac4feb05d2fe90bd99c989eeab9a\": container with ID starting with e035d50d5ae960a3ed6d8881755e550630c1ac4feb05d2fe90bd99c989eeab9a not found: ID does not exist" Oct 30 21:58:46.924823 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.924784 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmk66\" (UniqueName: \"kubernetes.io/projected/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-kube-api-access-cmk66\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.924823 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.924817 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.924823 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.924831 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.925098 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.924843 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.925098 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.924855 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 21:58:46.930591 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.930558 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf"] Oct 30 21:58:46.933122 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.933099 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/distinct-2-w6-ed088d-kserve-646df6bd65-62dsf"] Oct 30 21:58:46.963396 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.963358 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx"] Oct 30 21:58:46.967020 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.966985 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/premium-w6-ed088d-kserve-7669484547-xjwkx"] Oct 30 21:58:46.997715 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:46.997679 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb"] Oct 30 21:58:47.003121 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:47.003089 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/distinct-w6-ed088d-kserve-54f5f9f75d-424qb"] Oct 30 21:58:47.015762 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:47.015732 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" path="/var/lib/kubelet/pods/2ae4a221-e691-49b2-bafa-35e50ea1bead/volumes" Oct 30 21:58:47.016185 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:47.016172 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" path="/var/lib/kubelet/pods/d0cb8157-aaea-41e8-af1a-aef07b8a9bc4/volumes" Oct 30 21:58:47.016542 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:47.016530 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" path="/var/lib/kubelet/pods/d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d/volumes" Oct 30 21:58:47.044188 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:47.044152 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc"] Oct 30 21:58:47.046703 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:47.046666 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-ed088d/simulator-w6-ed088d-kserve-5bd47f6b5d-t9bbc"] Oct 30 21:58:49.014998 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:49.014960 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" path="/var/lib/kubelet/pods/1cc79717-77ce-4405-b102-ff904f2b345f/volumes" Oct 30 21:58:50.563494 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:50.563456 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:58:55.026501 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:55.026462 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v"] Oct 30 21:58:55.037240 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:55.037207 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aad3a63f-xl24v"] Oct 30 21:58:57.014803 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:58:57.014764 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="220c70bd-bd4c-4cb4-91f3-35604adcf782" path="/var/lib/kubelet/pods/220c70bd-bd4c-4cb4-91f3-35604adcf782/volumes" Oct 30 21:59:01.631325 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:59:01.631258 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 21:59:05.029714 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:59:05.029674 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92"] Oct 30 21:59:05.039214 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:59:05.039179 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb88ee49-pkv92"] Oct 30 21:59:07.015538 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:59:07.015492 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4792d7f-c239-4922-a6a5-82192ca0a807" path="/var/lib/kubelet/pods/d4792d7f-c239-4922-a6a5-82192ca0a807/volumes" Oct 30 21:59:07.484284 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:59:07.484246 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 21:59:19.454402 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:59:19.454365 4926 scope.go:122] "RemoveContainer" containerID="08f18fb9ce15168cfe99628bdf7eb17c9ad239e88093f35380074816e3144fcd" Oct 30 21:59:19.464683 ip-10-0-142-3 kubenswrapper[4926]: I1030 21:59:19.464652 4926 scope.go:122] "RemoveContainer" containerID="2b3a5cb9925ee58a598a0acc886a72e483e1b3d1a89a5d73f0b277a213c87c90" Oct 30 21:59:37.483433 ip-10-0-142-3 kubenswrapper[4926]: E1030 21:59:37.483390 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:00:07.483586 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:00:07.483544 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:00:13.927476 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:13.927445 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 22:00:13.931217 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:13.931189 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 22:00:17.128856 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:17.128808 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:00:19.519892 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:19.519850 4926 scope.go:122] "RemoveContainer" containerID="c258d4faa910dde3eb3a9622bb8b1c8e7a0f5bd94887413dae9840542234c725" Oct 30 22:00:19.530054 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:19.530015 4926 scope.go:122] "RemoveContainer" containerID="456fd2620f9a39da700c55b7151a0fd9cb9fa295b5d004bfd6c32c623a045b1d" Oct 30 22:00:19.539272 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:19.539250 4926 scope.go:122] "RemoveContainer" containerID="940130f85cba10400d86ddd10c173fba667a8044819259dab26a2192adb02767" Oct 30 22:00:19.548203 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:19.548181 4926 scope.go:122] "RemoveContainer" containerID="0d317aa4d3480ef1a500d24d0481b295deb585e8a132e63e7757674232e70b92" Oct 30 22:00:19.557175 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:19.557142 4926 scope.go:122] "RemoveContainer" containerID="6016ce8de4f5534f33436cad1d691852528df12dd86405f611739036e1a9baeb" Oct 30 22:00:19.566187 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:19.566157 4926 scope.go:122] "RemoveContainer" containerID="253fd78f6b35d546c7b1f41cf15e27c65764a05a9ba1310fac4223bbdfd07a4a" Oct 30 22:00:19.578817 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:19.578791 4926 scope.go:122] "RemoveContainer" containerID="c10835c1fecbe02f30b2a8844d43afddc7e7a31219118d9529a639369454e759" Oct 30 22:00:19.590794 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:19.590773 4926 scope.go:122] "RemoveContainer" containerID="adfa268de1b2df33080d71b9d147a6c778d979339951bfab5e8fced342e8e409" Oct 30 22:00:32.035746 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:32.035703 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de26f85c-bkrd7"] Oct 30 22:00:32.046859 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:32.046827 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de26f85c-bkrd7"] Oct 30 22:00:33.014616 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:33.014582 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4931c025-ff04-44cf-80ea-88c1d97df1e9" path="/var/lib/kubelet/pods/4931c025-ff04-44cf-80ea-88c1d97df1e9/volumes" Oct 30 22:00:33.033417 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:33.033384 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6"] Oct 30 22:00:33.044179 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:33.044130 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5dea2ab3-mpmm6"] Oct 30 22:00:34.138546 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:34.138508 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:00:35.014886 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:35.014849 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c19f7b-d721-4ffa-9b72-a671ea117191" path="/var/lib/kubelet/pods/d0c19f7b-d721-4ffa-9b72-a671ea117191/volumes" Oct 30 22:00:37.482975 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:00:37.482940 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:00:38.843816 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:38.843778 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:00:55.315140 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:55.315024 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:00:57.025529 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.025489 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw"] Oct 30 22:00:57.027104 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.026050 4926 cpu_manager.go:414] "containerMap: removing container" podUID="5285d5dc-80ad-40c6-b749-9c215955a4bd" containerName="revoke-keys" Oct 30 22:00:57.027104 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.026067 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5285d5dc-80ad-40c6-b749-9c215955a4bd" containerName="revoke-keys" Oct 30 22:00:57.027104 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.026089 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" Oct 30 22:00:57.027104 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.026097 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" Oct 30 22:00:57.027104 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.026109 4926 cpu_manager.go:414] "containerMap: removing container" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" Oct 30 22:00:57.027104 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.026126 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" Oct 30 22:00:57.027104 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.026136 4926 cpu_manager.go:414] "containerMap: removing container" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" Oct 30 22:00:57.027104 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.026144 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" Oct 30 22:00:57.027104 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.026159 4926 cpu_manager.go:414] "containerMap: removing container" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" Oct 30 22:00:57.027104 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.026166 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" Oct 30 22:00:57.027104 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.026277 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c3ba5c-c18e-46bd-ae0a-b2eb12e1b72d" containerName="main" Oct 30 22:00:57.027104 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.026289 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cc79717-77ce-4405-b102-ff904f2b345f" containerName="main" Oct 30 22:00:57.027104 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.026298 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae4a221-e691-49b2-bafa-35e50ea1bead" containerName="main" Oct 30 22:00:57.027104 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.026308 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0cb8157-aaea-41e8-af1a-aef07b8a9bc4" containerName="main" Oct 30 22:00:57.027104 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.026326 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="5285d5dc-80ad-40c6-b749-9c215955a4bd" containerName="revoke-keys" Oct 30 22:00:57.038157 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.038131 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.068985 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.068939 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw"] Oct 30 22:00:57.146221 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.146182 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed448351-dac3-4d81-9e30-bce9475607eb-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.146405 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.146232 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed448351-dac3-4d81-9e30-bce9475607eb-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.146405 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.146328 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed448351-dac3-4d81-9e30-bce9475607eb-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.146489 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.146411 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed448351-dac3-4d81-9e30-bce9475607eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.146489 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.146477 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed448351-dac3-4d81-9e30-bce9475607eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.146560 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.146506 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed448351-dac3-4d81-9e30-bce9475607eb-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.146560 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.146545 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed448351-dac3-4d81-9e30-bce9475607eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.146635 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.146581 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s6cg\" (UniqueName: \"kubernetes.io/projected/ed448351-dac3-4d81-9e30-bce9475607eb-kube-api-access-9s6cg\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.146701 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.146675 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed448351-dac3-4d81-9e30-bce9475607eb-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.247900 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.247860 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed448351-dac3-4d81-9e30-bce9475607eb-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.248139 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.247914 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed448351-dac3-4d81-9e30-bce9475607eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.248139 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.247947 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed448351-dac3-4d81-9e30-bce9475607eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.248139 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.247974 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed448351-dac3-4d81-9e30-bce9475607eb-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.248139 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.248001 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed448351-dac3-4d81-9e30-bce9475607eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.248394 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.248171 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s6cg\" (UniqueName: \"kubernetes.io/projected/ed448351-dac3-4d81-9e30-bce9475607eb-kube-api-access-9s6cg\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.248394 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.248249 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed448351-dac3-4d81-9e30-bce9475607eb-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.248394 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.248281 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed448351-dac3-4d81-9e30-bce9475607eb-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.248394 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.248329 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed448351-dac3-4d81-9e30-bce9475607eb-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.248394 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.248343 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed448351-dac3-4d81-9e30-bce9475607eb-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.248763 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.248740 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed448351-dac3-4d81-9e30-bce9475607eb-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.248846 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.248741 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed448351-dac3-4d81-9e30-bce9475607eb-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.248956 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.248941 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed448351-dac3-4d81-9e30-bce9475607eb-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.248995 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.248957 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed448351-dac3-4d81-9e30-bce9475607eb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.250475 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.250445 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed448351-dac3-4d81-9e30-bce9475607eb-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.250578 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.250561 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed448351-dac3-4d81-9e30-bce9475607eb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.267649 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.267621 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed448351-dac3-4d81-9e30-bce9475607eb-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.267761 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.267658 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s6cg\" (UniqueName: \"kubernetes.io/projected/ed448351-dac3-4d81-9e30-bce9475607eb-kube-api-access-9s6cg\") pod \"maas-default-gateway-openshift-default-57f78bcd55-rh5rw\" (UID: \"ed448351-dac3-4d81-9e30-bce9475607eb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.352114 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.352001 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:57.680773 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.680735 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw"] Oct 30 22:00:57.686888 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.686314 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:00:57.686888 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.686404 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:00:57.686888 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.686450 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:00:57.775336 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.775186 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" event={"ID":"ed448351-dac3-4d81-9e30-bce9475607eb","Type":"ContainerStarted","Data":"1ef46b059a78a8b35dab758456d196a9e7279af63abb00309fdd18661a9cdb51"} Oct 30 22:00:57.775336 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.775256 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" event={"ID":"ed448351-dac3-4d81-9e30-bce9475607eb","Type":"ContainerStarted","Data":"942398f5ac36f83a979474d5df9761cdd5496f81c47e7fab2ebfcffbe9ef3c03"} Oct 30 22:00:57.822925 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:57.821946 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" podStartSLOduration=0.821930397 podStartE2EDuration="821.930397ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:57.821458621 +0000 UTC m=+3644.942532370" watchObservedRunningTime="2026-10-30 22:00:57.821930397 +0000 UTC m=+3644.943004212" Oct 30 22:00:58.352515 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:58.352479 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:00:58.354616 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:58.354578 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rh5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Oct 30 22:00:58.354783 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:58.354635 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" podUID="ed448351-dac3-4d81-9e30-bce9475607eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Oct 30 22:00:59.353241 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:59.353205 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rh5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Oct 30 22:00:59.353674 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:00:59.353261 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" podUID="ed448351-dac3-4d81-9e30-bce9475607eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Oct 30 22:01:00.352540 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:00.352502 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rh5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Oct 30 22:01:00.352744 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:00.352561 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" podUID="ed448351-dac3-4d81-9e30-bce9475607eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Oct 30 22:01:00.636399 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:00.636310 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:01:01.352584 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:01.352549 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rh5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Oct 30 22:01:01.352797 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:01.352600 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" podUID="ed448351-dac3-4d81-9e30-bce9475607eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Oct 30 22:01:02.353380 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:02.353336 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rh5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Oct 30 22:01:02.353804 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:02.353396 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" podUID="ed448351-dac3-4d81-9e30-bce9475607eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Oct 30 22:01:03.352694 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:03.352642 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rh5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Oct 30 22:01:03.352898 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:03.352724 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" podUID="ed448351-dac3-4d81-9e30-bce9475607eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Oct 30 22:01:04.352753 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:04.352714 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rh5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" start-of-body= Oct 30 22:01:04.353269 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:04.352777 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" podUID="ed448351-dac3-4d81-9e30-bce9475607eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": dial tcp 10.132.1.70:15021: connect: connection refused" Oct 30 22:01:06.353202 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:06.353149 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rh5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:01:06.353794 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:06.353227 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" podUID="ed448351-dac3-4d81-9e30-bce9475607eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:01:07.395319 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:07.395262 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rh5rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:01:07.395883 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:07.395357 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" podUID="ed448351-dac3-4d81-9e30-bce9475607eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:01:07.487779 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:01:07.487731 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:01:08.124734 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:08.124698 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:01:08.125232 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:08.125214 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:01:08.125936 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:08.125909 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" Oct 30 22:01:09.901511 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:09.901474 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:01:12.097415 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.097377 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4"] Oct 30 22:01:12.109512 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.109459 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.138941 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.138898 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4"] Oct 30 22:01:12.235602 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.235275 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36be1c6a-ce91-4582-afef-56c4d74c0335-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.235602 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.235356 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36be1c6a-ce91-4582-afef-56c4d74c0335-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.235602 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.235401 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36be1c6a-ce91-4582-afef-56c4d74c0335-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.235602 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.235491 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36be1c6a-ce91-4582-afef-56c4d74c0335-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.235602 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.235547 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36be1c6a-ce91-4582-afef-56c4d74c0335-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.235970 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.235631 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36be1c6a-ce91-4582-afef-56c4d74c0335-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.235970 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.235677 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qlpk\" (UniqueName: \"kubernetes.io/projected/36be1c6a-ce91-4582-afef-56c4d74c0335-kube-api-access-5qlpk\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.235970 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.235699 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36be1c6a-ce91-4582-afef-56c4d74c0335-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.235970 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.235852 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36be1c6a-ce91-4582-afef-56c4d74c0335-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.337343 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.337298 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qlpk\" (UniqueName: \"kubernetes.io/projected/36be1c6a-ce91-4582-afef-56c4d74c0335-kube-api-access-5qlpk\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.337343 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.337345 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36be1c6a-ce91-4582-afef-56c4d74c0335-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.337560 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.337382 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36be1c6a-ce91-4582-afef-56c4d74c0335-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.337560 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.337408 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36be1c6a-ce91-4582-afef-56c4d74c0335-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.337560 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.337423 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36be1c6a-ce91-4582-afef-56c4d74c0335-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.337560 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.337439 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36be1c6a-ce91-4582-afef-56c4d74c0335-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.337560 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.337460 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36be1c6a-ce91-4582-afef-56c4d74c0335-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.337560 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.337474 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36be1c6a-ce91-4582-afef-56c4d74c0335-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.337560 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.337518 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36be1c6a-ce91-4582-afef-56c4d74c0335-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.337865 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.337849 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36be1c6a-ce91-4582-afef-56c4d74c0335-credential-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.338402 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.338372 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36be1c6a-ce91-4582-afef-56c4d74c0335-workload-certs\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.338545 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.338488 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36be1c6a-ce91-4582-afef-56c4d74c0335-istio-data\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.338545 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.338505 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36be1c6a-ce91-4582-afef-56c4d74c0335-workload-socket\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.338807 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.338785 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36be1c6a-ce91-4582-afef-56c4d74c0335-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.340143 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.340123 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36be1c6a-ce91-4582-afef-56c4d74c0335-istio-envoy\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.340539 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.340516 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36be1c6a-ce91-4582-afef-56c4d74c0335-istio-podinfo\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.356094 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.356024 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36be1c6a-ce91-4582-afef-56c4d74c0335-istio-token\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.356183 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.356147 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qlpk\" (UniqueName: \"kubernetes.io/projected/36be1c6a-ce91-4582-afef-56c4d74c0335-kube-api-access-5qlpk\") pod \"maas-default-gateway-openshift-default-57f78bcd55-bpsh4\" (UID: \"36be1c6a-ce91-4582-afef-56c4d74c0335\") " pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.450662 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.450621 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:12.818972 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.818939 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4"] Oct 30 22:01:12.822619 ip-10-0-142-3 kubenswrapper[4926]: W1030 22:01:12.822582 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36be1c6a_ce91_4582_afef_56c4d74c0335.slice/crio-bfbf30367c084b0b56cafd44c3c3b0e077e16516622016458cfa31c5a52a7227 WatchSource:0}: Error finding container bfbf30367c084b0b56cafd44c3c3b0e077e16516622016458cfa31c5a52a7227: Status 404 returned error can't find the container with id bfbf30367c084b0b56cafd44c3c3b0e077e16516622016458cfa31c5a52a7227 Oct 30 22:01:12.825417 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.825372 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:01:12.825557 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.825471 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:01:12.825557 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.825521 4926 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 22:01:12.917677 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.917634 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" event={"ID":"36be1c6a-ce91-4582-afef-56c4d74c0335","Type":"ContainerStarted","Data":"fd53ead806be7281787178bafa29ff1ad9520e22288468baa80ef7a0915918e2"} Oct 30 22:01:12.917890 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.917708 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" event={"ID":"36be1c6a-ce91-4582-afef-56c4d74c0335","Type":"ContainerStarted","Data":"bfbf30367c084b0b56cafd44c3c3b0e077e16516622016458cfa31c5a52a7227"} Oct 30 22:01:12.957544 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:12.957487 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" podStartSLOduration=0.957474113 podStartE2EDuration="957.474113ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:12.957306568 +0000 UTC m=+3660.078380315" watchObservedRunningTime="2026-10-30 22:01:12.957474113 +0000 UTC m=+3660.078547897" Oct 30 22:01:13.451727 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:13.451686 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:13.453487 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:13.453460 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bpsh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body= Oct 30 22:01:13.453542 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:13.453512 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" podUID="36be1c6a-ce91-4582-afef-56c4d74c0335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" Oct 30 22:01:14.451343 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:14.451306 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bpsh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body= Oct 30 22:01:14.451543 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:14.451366 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" podUID="36be1c6a-ce91-4582-afef-56c4d74c0335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" Oct 30 22:01:15.451966 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:15.451921 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bpsh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body= Oct 30 22:01:15.452413 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:15.451995 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" podUID="36be1c6a-ce91-4582-afef-56c4d74c0335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" Oct 30 22:01:16.451854 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:16.451806 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bpsh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body= Oct 30 22:01:16.452072 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:16.451879 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" podUID="36be1c6a-ce91-4582-afef-56c4d74c0335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" Oct 30 22:01:17.451761 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:17.451726 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bpsh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body= Oct 30 22:01:17.451964 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:17.451783 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" podUID="36be1c6a-ce91-4582-afef-56c4d74c0335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" Oct 30 22:01:18.451200 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:18.451164 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bpsh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body= Oct 30 22:01:18.451630 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:18.451218 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" podUID="36be1c6a-ce91-4582-afef-56c4d74c0335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" Oct 30 22:01:19.451466 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:19.451428 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bpsh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" start-of-body= Oct 30 22:01:19.451883 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:19.451484 4926 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" podUID="36be1c6a-ce91-4582-afef-56c4d74c0335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": dial tcp 10.132.1.71:15021: connect: connection refused" Oct 30 22:01:19.647346 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:19.647311 4926 scope.go:122] "RemoveContainer" containerID="55b80b19f16d3b086b9f5323cce7d8055fa518cac4bd72aa13edd2c40dd302ba" Oct 30 22:01:19.657364 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:19.657343 4926 scope.go:122] "RemoveContainer" containerID="2a7644386c0bb92618913f3ef4783dcc283536de6c189861a7ef47c8363dc6c2" Oct 30 22:01:19.666731 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:19.666703 4926 scope.go:122] "RemoveContainer" containerID="2ce1d6f5386247b0be5e6a7fc463ac02fdd0487f9a29bcb4c067bec775fe335f" Oct 30 22:01:19.677796 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:19.677771 4926 scope.go:122] "RemoveContainer" containerID="3ea7750fada069e75036b6e7fd34d480ba23a2222a15282cb36b732227ab742c" Oct 30 22:01:19.689104 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:19.689018 4926 scope.go:122] "RemoveContainer" containerID="1f687370f4b3a4ae555a54472fd8b0d94f4fb89282345330fe348f5f0286a9ec" Oct 30 22:01:19.699724 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:19.699698 4926 scope.go:122] "RemoveContainer" containerID="56db70cd8511776a651eeb315d5c8748db5668f525f583b21010d115befcc70d" Oct 30 22:01:19.708645 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:19.708623 4926 scope.go:122] "RemoveContainer" containerID="7fe1675698a12817ae72aa0f335e2da13b46bfa827014cfa35d1a1cb2f5d839d" Oct 30 22:01:21.082127 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:21.082089 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:21.082653 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:21.082483 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:21.083336 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:21.083314 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" Oct 30 22:01:22.028813 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:22.028773 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5"] Oct 30 22:01:22.040261 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:22.040205 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab02655b-j5zv5"] Oct 30 22:01:22.043745 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:22.043710 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf"] Oct 30 22:01:22.047890 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:22.047865 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd8173a-lt8nf"] Oct 30 22:01:23.014731 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:23.014693 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="084dee41-d4ad-4a17-844d-34aa27886965" path="/var/lib/kubelet/pods/084dee41-d4ad-4a17-844d-34aa27886965/volumes" Oct 30 22:01:23.015181 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:23.015165 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fea7423-7d7a-453f-aa15-b552ba9f43fd" path="/var/lib/kubelet/pods/8fea7423-7d7a-453f-aa15-b552ba9f43fd/volumes" Oct 30 22:01:28.027659 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:28.027616 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh"] Oct 30 22:01:28.038746 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:28.038709 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-809d8e0c-v2qlh"] Oct 30 22:01:28.040964 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:28.040943 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6"] Oct 30 22:01:28.042854 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:28.042831 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125b01f8-whcv6"] Oct 30 22:01:29.014883 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:29.014843 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="015125ae-8075-4e8f-b53f-7e1a0062be96" path="/var/lib/kubelet/pods/015125ae-8075-4e8f-b53f-7e1a0062be96/volumes" Oct 30 22:01:29.015356 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:29.015337 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c88fb565-e4b8-4389-852e-68b980559848" path="/var/lib/kubelet/pods/c88fb565-e4b8-4389-852e-68b980559848/volumes" Oct 30 22:01:34.652808 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:34.652773 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:01:37.134237 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:37.134179 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-bpsh4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:01:37.134950 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:37.134910 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-bpsh4" podUID="36be1c6a-ce91-4582-afef-56c4d74c0335" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:01:37.533834 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:01:37.533794 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:01:45.113578 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:45.113541 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:01:50.048599 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:01:50.048564 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:02:06.545565 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:06.545526 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:02:07.483740 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:02:07.483706 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:02:11.836598 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:11.836556 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:02:19.762438 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:19.762396 4926 scope.go:122] "RemoveContainer" containerID="c2c3d173253d43de317b4d7a7b9b91a19c653f7e16c4a393401a929a717ec8e2" Oct 30 22:02:19.772590 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:19.772566 4926 scope.go:122] "RemoveContainer" containerID="f4023f8d251f50ec42a8d88464fa9ef8f15478d2328e0cff3892e5290ab24c5a" Oct 30 22:02:19.781558 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:19.781535 4926 scope.go:122] "RemoveContainer" containerID="338db3482bf099d6b1eb6df1a06a97edf8acc7806e40ab6915aea9a005c14319" Oct 30 22:02:19.790366 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:19.790347 4926 scope.go:122] "RemoveContainer" containerID="1f936626315eabd5982f8dc3aed39bae42c2531b86e7054f63bb5e8636549625" Oct 30 22:02:19.798877 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:19.798859 4926 scope.go:122] "RemoveContainer" containerID="605d123d66bf4a7b7cbd11d77ffc26e8096b26a6112361b80bb4b79d1728acbf" Oct 30 22:02:19.807787 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:19.807767 4926 scope.go:122] "RemoveContainer" containerID="8b06432ae85232e930c9d64301210f515fde2dc9b467a32a7ee1ab3a2abc8447" Oct 30 22:02:19.815907 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:19.815887 4926 scope.go:122] "RemoveContainer" containerID="3f3cf194fdc0f8926ec908e5aabfba0dccaa36174e75d05186d303026fbd3744" Oct 30 22:02:19.824668 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:19.824642 4926 scope.go:122] "RemoveContainer" containerID="66f9d5f1ad029a8ea28bf61da3ad2dadbeb7c95c84ad5a794db0df4136da576f" Oct 30 22:02:19.833386 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:19.833364 4926 scope.go:122] "RemoveContainer" containerID="27c51ee2fed58c25039984d2ee7d66316d6886a721eb705c377f6f0987243fed" Oct 30 22:02:20.029079 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:20.028962 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c54e7e6b-4l64w"] Oct 30 22:02:20.040058 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:20.040011 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg"] Oct 30 22:02:20.042333 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:20.042308 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c54e7e6b-4l64w"] Oct 30 22:02:20.044150 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:20.044128 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d5f588f-2pvcg"] Oct 30 22:02:21.014551 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:21.014520 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cae23f38-df9f-443a-bf6b-aa20e5e365cd" path="/var/lib/kubelet/pods/cae23f38-df9f-443a-bf6b-aa20e5e365cd/volumes" Oct 30 22:02:21.014960 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:21.014946 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e89d855c-7de9-409a-9644-474afb490832" path="/var/lib/kubelet/pods/e89d855c-7de9-409a-9644-474afb490832/volumes" Oct 30 22:02:27.999932 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:27.999849 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:02:33.384208 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:33.384160 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:02:37.484134 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:02:37.484092 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:02:50.111418 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:50.111368 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:02:58.026752 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:58.026709 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d70dc77b-77p7f"] Oct 30 22:02:58.037898 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:58.037862 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d70dc77b-77p7f"] Oct 30 22:02:58.039993 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:58.039966 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd92aef-n5hh8"] Oct 30 22:02:58.041809 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:58.041786 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdd92aef-n5hh8"] Oct 30 22:02:59.014933 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:59.014897 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fd13f33-da83-4b40-b2ed-102936728505" path="/var/lib/kubelet/pods/1fd13f33-da83-4b40-b2ed-102936728505/volumes" Oct 30 22:02:59.015372 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:02:59.015357 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90bd49e9-63e3-4e33-a644-4441e33f5663" path="/var/lib/kubelet/pods/90bd49e9-63e3-4e33-a644-4441e33f5663/volumes" Oct 30 22:03:06.158258 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:03:06.158214 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:03:07.483158 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:03:07.483118 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:03:16.832450 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:03:16.832412 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:03:19.893539 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:03:19.893494 4926 scope.go:122] "RemoveContainer" containerID="3f235a3876f505c8bf9caed20e205281bd2408b1d729252af13afc5d8555087b" Oct 30 22:03:19.903226 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:03:19.903203 4926 scope.go:122] "RemoveContainer" containerID="39c01caad133359a2535fff23a6bf1665a37bd50e6fc93c5205e449d7c4c9c5e" Oct 30 22:03:19.912766 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:03:19.912739 4926 scope.go:122] "RemoveContainer" containerID="ead6b8974fd43ad35b4c9691fc0fb442a41540bb43c8192c13c756f95025b147" Oct 30 22:03:19.923105 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:03:19.923080 4926 scope.go:122] "RemoveContainer" containerID="d83b4be314076531ebfc8bd2f28f85853b2e4264d7989c70f4999b3e83c058d4" Oct 30 22:03:19.931499 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:03:19.931470 4926 scope.go:122] "RemoveContainer" containerID="102999b4aec62309d83df6b4bb8077bbb6de23025185e029bd25cd1be2b18b17" Oct 30 22:03:19.939531 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:03:19.939509 4926 scope.go:122] "RemoveContainer" containerID="ff71d4878b12098011cd48588363343149659ca75ad3e4d51190b6a66ed1f074" Oct 30 22:03:19.947521 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:03:19.947503 4926 scope.go:122] "RemoveContainer" containerID="0760f5cc38f4f34116249ec423418145932d2d4d7cc531420af75dcc7e6a0621" Oct 30 22:03:19.955894 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:03:19.955875 4926 scope.go:122] "RemoveContainer" containerID="1e377cb55c705bd6b7021bdb000934f274c87ca5142b873ce2b073a9a1e76b69" Oct 30 22:03:19.964069 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:03:19.964027 4926 scope.go:122] "RemoveContainer" containerID="a1e4b9e381dd938f5f131b186384c9711484929e13da541ba4d7a6a274990120" Oct 30 22:03:19.971873 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:03:19.971854 4926 scope.go:122] "RemoveContainer" containerID="716c7df33597e2a5c22c23f352f087f5512cdc4c558154a64d0057b1f48a5389" Oct 30 22:03:22.821579 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:03:22.821545 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:03:37.484257 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:03:37.484217 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:03:41.030176 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:03:41.030141 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l"] Oct 30 22:03:41.042642 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:03:41.042607 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bf396ba-5fx8l"] Oct 30 22:03:43.014867 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:03:43.014829 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5285d5dc-80ad-40c6-b749-9c215955a4bd" path="/var/lib/kubelet/pods/5285d5dc-80ad-40c6-b749-9c215955a4bd/volumes" Oct 30 22:03:48.228256 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:03:48.228213 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:04:06.450300 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:04:06.450253 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:04:07.486003 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:04:07.485960 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:04:09.181377 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:04:09.181333 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rh5rw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:04:09.182550 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:04:09.182513 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" podUID="ed448351-dac3-4d81-9e30-bce9475607eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:04:16.863578 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:04:16.863534 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:04:20.033309 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:04:20.033272 4926 scope.go:122] "RemoveContainer" containerID="5b155f3fd39708cf9da7e27da7f0a04667550fcc11127226ce70a987a7e15684" Oct 30 22:04:20.045348 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:04:20.045320 4926 scope.go:122] "RemoveContainer" containerID="af6e2f94f5cb14aa266b017de366b14778b0c011662810f28c353fc537f0cd41" Oct 30 22:04:20.054360 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:04:20.054336 4926 scope.go:122] "RemoveContainer" containerID="1ed8738b729ed9cb467df3980329faf4ea7c9d91419c3df816a92260bcb2b615" Oct 30 22:04:20.063140 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:04:20.063108 4926 scope.go:122] "RemoveContainer" containerID="198b927f1603c08c3d75b5fc6960d241b966ecf62ecf137910ffa216202e8fd5" Oct 30 22:04:20.074218 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:04:20.074188 4926 scope.go:122] "RemoveContainer" containerID="4c9cffb1cade8459d7b90c157b7edf61bf717db69b17056c447bc694d25859b7" Oct 30 22:04:37.483473 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:04:37.483431 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:04:50.072626 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:04:50.072579 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:05:05.250863 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:05.250817 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:05:07.484438 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:05:07.484402 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:05:13.964925 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:13.964896 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 22:05:13.971080 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:13.971048 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 22:05:20.117182 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:20.117145 4926 scope.go:122] "RemoveContainer" containerID="91ca1091e2e8a2715164e9d04bf02286a3689f0f1ce6abd17d6774f84b87717a" Oct 30 22:05:20.328806 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:20.328766 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:05:34.001644 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:34.001552 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ebd54dd-wd6vb"] Oct 30 22:05:34.006113 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:34.006092 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebd54dd-wd6vb" Oct 30 22:05:34.020828 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:34.020803 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:05:34.032149 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:34.032117 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:05:34.032308 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:34.032177 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:05:34.032308 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:34.032194 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 22:05:34.034317 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:34.034293 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ebd54dd-wd6vb"] Oct 30 22:05:34.155119 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:34.155081 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ef08bbd-125f-4fcf-a445-cd1fd966c3a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ebd54dd-wd6vb\" (UID: \"1ef08bbd-125f-4fcf-a445-cd1fd966c3a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebd54dd-wd6vb" Oct 30 22:05:34.155352 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:34.155228 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ef08bbd-125f-4fcf-a445-cd1fd966c3a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ebd54dd-wd6vb\" (UID: \"1ef08bbd-125f-4fcf-a445-cd1fd966c3a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebd54dd-wd6vb" Oct 30 22:05:34.155440 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:34.155357 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb2tm\" (UniqueName: \"kubernetes.io/projected/1ef08bbd-125f-4fcf-a445-cd1fd966c3a0-kube-api-access-tb2tm\") pod \"maas-api-revoke-sub-4ebd54dd-wd6vb\" (UID: \"1ef08bbd-125f-4fcf-a445-cd1fd966c3a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebd54dd-wd6vb" Oct 30 22:05:34.256860 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:34.256764 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ef08bbd-125f-4fcf-a445-cd1fd966c3a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ebd54dd-wd6vb\" (UID: \"1ef08bbd-125f-4fcf-a445-cd1fd966c3a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebd54dd-wd6vb" Oct 30 22:05:34.256860 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:34.256836 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ef08bbd-125f-4fcf-a445-cd1fd966c3a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ebd54dd-wd6vb\" (UID: \"1ef08bbd-125f-4fcf-a445-cd1fd966c3a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebd54dd-wd6vb" Oct 30 22:05:34.257094 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:34.256889 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb2tm\" (UniqueName: \"kubernetes.io/projected/1ef08bbd-125f-4fcf-a445-cd1fd966c3a0-kube-api-access-tb2tm\") pod \"maas-api-revoke-sub-4ebd54dd-wd6vb\" (UID: \"1ef08bbd-125f-4fcf-a445-cd1fd966c3a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebd54dd-wd6vb" Oct 30 22:05:34.257644 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:34.257622 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ef08bbd-125f-4fcf-a445-cd1fd966c3a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ebd54dd-wd6vb\" (UID: \"1ef08bbd-125f-4fcf-a445-cd1fd966c3a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebd54dd-wd6vb" Oct 30 22:05:34.259190 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:34.259161 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ef08bbd-125f-4fcf-a445-cd1fd966c3a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ebd54dd-wd6vb\" (UID: \"1ef08bbd-125f-4fcf-a445-cd1fd966c3a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebd54dd-wd6vb" Oct 30 22:05:34.276210 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:34.276175 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb2tm\" (UniqueName: \"kubernetes.io/projected/1ef08bbd-125f-4fcf-a445-cd1fd966c3a0-kube-api-access-tb2tm\") pod \"maas-api-revoke-sub-4ebd54dd-wd6vb\" (UID: \"1ef08bbd-125f-4fcf-a445-cd1fd966c3a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebd54dd-wd6vb" Oct 30 22:05:34.319648 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:34.319612 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebd54dd-wd6vb" Oct 30 22:05:34.615553 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:34.615526 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ebd54dd-wd6vb"] Oct 30 22:05:34.618376 ip-10-0-142-3 kubenswrapper[4926]: W1030 22:05:34.618343 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef08bbd_125f_4fcf_a445_cd1fd966c3a0.slice/crio-33a310df79dfb5912dd39fcc83941c269ebde7e4d44d67babe38d714e0e568a4 WatchSource:0}: Error finding container 33a310df79dfb5912dd39fcc83941c269ebde7e4d44d67babe38d714e0e568a4: Status 404 returned error can't find the container with id 33a310df79dfb5912dd39fcc83941c269ebde7e4d44d67babe38d714e0e568a4 Oct 30 22:05:34.634284 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:34.634253 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebd54dd-wd6vb" event={"ID":"1ef08bbd-125f-4fcf-a445-cd1fd966c3a0","Type":"ContainerStarted","Data":"33a310df79dfb5912dd39fcc83941c269ebde7e4d44d67babe38d714e0e568a4"} Oct 30 22:05:35.641610 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:35.641519 4926 generic.go:356] "Generic (PLEG): container finished" podID="1ef08bbd-125f-4fcf-a445-cd1fd966c3a0" containerID="f53da5fb9b211421e1a982670266cb6a190fee901870b7d69f093b7522e8018b" exitCode=0 Oct 30 22:05:35.641976 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:35.641606 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebd54dd-wd6vb" event={"ID":"1ef08bbd-125f-4fcf-a445-cd1fd966c3a0","Type":"ContainerDied","Data":"f53da5fb9b211421e1a982670266cb6a190fee901870b7d69f093b7522e8018b"} Oct 30 22:05:36.780783 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:36.780751 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebd54dd-wd6vb" Oct 30 22:05:36.881647 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:36.881610 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef08bbd-125f-4fcf-a445-cd1fd966c3a0-kube-api-access-tb2tm\" (UniqueName: \"kubernetes.io/projected/1ef08bbd-125f-4fcf-a445-cd1fd966c3a0-kube-api-access-tb2tm\") pod \"1ef08bbd-125f-4fcf-a445-cd1fd966c3a0\" (UID: \"1ef08bbd-125f-4fcf-a445-cd1fd966c3a0\") " Oct 30 22:05:36.881862 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:36.881691 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ef08bbd-125f-4fcf-a445-cd1fd966c3a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ef08bbd-125f-4fcf-a445-cd1fd966c3a0-maas-api-xks-ca\") pod \"1ef08bbd-125f-4fcf-a445-cd1fd966c3a0\" (UID: \"1ef08bbd-125f-4fcf-a445-cd1fd966c3a0\") " Oct 30 22:05:36.881862 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:36.881800 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ef08bbd-125f-4fcf-a445-cd1fd966c3a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ef08bbd-125f-4fcf-a445-cd1fd966c3a0-maas-api-openshift-ca\") pod \"1ef08bbd-125f-4fcf-a445-cd1fd966c3a0\" (UID: \"1ef08bbd-125f-4fcf-a445-cd1fd966c3a0\") " Oct 30 22:05:36.882201 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:36.882169 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef08bbd-125f-4fcf-a445-cd1fd966c3a0-maas-api-openshift-ca" pod "1ef08bbd-125f-4fcf-a445-cd1fd966c3a0" (UID: "1ef08bbd-125f-4fcf-a445-cd1fd966c3a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:36.883790 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:36.883762 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ef08bbd-125f-4fcf-a445-cd1fd966c3a0-maas-api-xks-ca" pod "1ef08bbd-125f-4fcf-a445-cd1fd966c3a0" (UID: "1ef08bbd-125f-4fcf-a445-cd1fd966c3a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:36.883884 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:36.883798 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef08bbd-125f-4fcf-a445-cd1fd966c3a0-kube-api-access-tb2tm" pod "1ef08bbd-125f-4fcf-a445-cd1fd966c3a0" (UID: "1ef08bbd-125f-4fcf-a445-cd1fd966c3a0"). InnerVolumeSpecName "kube-api-access-tb2tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:36.982568 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:36.982527 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ef08bbd-125f-4fcf-a445-cd1fd966c3a0-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:05:36.982568 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:36.982561 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ef08bbd-125f-4fcf-a445-cd1fd966c3a0-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:05:36.982568 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:36.982571 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb2tm\" (UniqueName: \"kubernetes.io/projected/1ef08bbd-125f-4fcf-a445-cd1fd966c3a0-kube-api-access-tb2tm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:05:37.483575 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:05:37.483536 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:05:37.657979 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:37.657942 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebd54dd-wd6vb" event={"ID":"1ef08bbd-125f-4fcf-a445-cd1fd966c3a0","Type":"ContainerDied","Data":"33a310df79dfb5912dd39fcc83941c269ebde7e4d44d67babe38d714e0e568a4"} Oct 30 22:05:37.658185 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:37.657994 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33a310df79dfb5912dd39fcc83941c269ebde7e4d44d67babe38d714e0e568a4" Oct 30 22:05:37.658185 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:37.657949 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ebd54dd-wd6vb" Oct 30 22:05:45.372913 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:45.372871 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x"] Oct 30 22:05:45.373534 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:45.373512 4926 cpu_manager.go:414] "containerMap: removing container" podUID="1ef08bbd-125f-4fcf-a445-cd1fd966c3a0" containerName="revoke-keys" Oct 30 22:05:45.373633 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:45.373537 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef08bbd-125f-4fcf-a445-cd1fd966c3a0" containerName="revoke-keys" Oct 30 22:05:45.373703 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:45.373641 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef08bbd-125f-4fcf-a445-cd1fd966c3a0" containerName="revoke-keys" Oct 30 22:05:45.377153 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:45.377132 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x" Oct 30 22:05:45.392076 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:45.392044 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:05:45.402962 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:45.402918 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:05:45.403247 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:45.403220 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:05:45.403377 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:45.403311 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 22:05:45.406766 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:45.406734 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x"] Oct 30 22:05:45.460908 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:45.460872 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29140a1f-6928-4042-a96b-d74a0db8e138-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x\" (UID: \"29140a1f-6928-4042-a96b-d74a0db8e138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x" Oct 30 22:05:45.461123 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:45.460915 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb98b\" (UniqueName: \"kubernetes.io/projected/29140a1f-6928-4042-a96b-d74a0db8e138-kube-api-access-vb98b\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x\" (UID: \"29140a1f-6928-4042-a96b-d74a0db8e138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x" Oct 30 22:05:45.461123 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:45.461087 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29140a1f-6928-4042-a96b-d74a0db8e138-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x\" (UID: \"29140a1f-6928-4042-a96b-d74a0db8e138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x" Oct 30 22:05:45.562015 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:45.561972 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29140a1f-6928-4042-a96b-d74a0db8e138-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x\" (UID: \"29140a1f-6928-4042-a96b-d74a0db8e138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x" Oct 30 22:05:45.562233 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:45.562025 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29140a1f-6928-4042-a96b-d74a0db8e138-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x\" (UID: \"29140a1f-6928-4042-a96b-d74a0db8e138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x" Oct 30 22:05:45.562233 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:45.562086 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb98b\" (UniqueName: \"kubernetes.io/projected/29140a1f-6928-4042-a96b-d74a0db8e138-kube-api-access-vb98b\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x\" (UID: \"29140a1f-6928-4042-a96b-d74a0db8e138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x" Oct 30 22:05:45.562695 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:45.562668 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29140a1f-6928-4042-a96b-d74a0db8e138-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x\" (UID: \"29140a1f-6928-4042-a96b-d74a0db8e138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x" Oct 30 22:05:45.564579 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:45.564549 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29140a1f-6928-4042-a96b-d74a0db8e138-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x\" (UID: \"29140a1f-6928-4042-a96b-d74a0db8e138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x" Oct 30 22:05:45.580814 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:45.580789 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb98b\" (UniqueName: \"kubernetes.io/projected/29140a1f-6928-4042-a96b-d74a0db8e138-kube-api-access-vb98b\") pod \"maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x\" (UID: \"29140a1f-6928-4042-a96b-d74a0db8e138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x" Oct 30 22:05:45.690915 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:45.690879 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x" Oct 30 22:05:45.992145 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:45.992105 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x"] Oct 30 22:05:45.999135 ip-10-0-142-3 kubenswrapper[4926]: W1030 22:05:45.999093 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29140a1f_6928_4042_a96b_d74a0db8e138.slice/crio-0c6c2040f851ca4a4c014a947a169f8322a2ec6f99003b21e581c2c65fe99129 WatchSource:0}: Error finding container 0c6c2040f851ca4a4c014a947a169f8322a2ec6f99003b21e581c2c65fe99129: Status 404 returned error can't find the container with id 0c6c2040f851ca4a4c014a947a169f8322a2ec6f99003b21e581c2c65fe99129 Oct 30 22:05:46.548943 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:46.548858 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:05:46.712210 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:46.712162 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x" event={"ID":"29140a1f-6928-4042-a96b-d74a0db8e138","Type":"ContainerStarted","Data":"52fbb9b72166ad418113d20bafc6381b41c0c67ce633f4ca283ba9a91eef7885"} Oct 30 22:05:46.712426 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:46.712250 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x" event={"ID":"29140a1f-6928-4042-a96b-d74a0db8e138","Type":"ContainerStarted","Data":"0c6c2040f851ca4a4c014a947a169f8322a2ec6f99003b21e581c2c65fe99129"} Oct 30 22:05:46.752747 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:46.752686 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x" podStartSLOduration=1.75266442 podStartE2EDuration="1.75266442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:46.752277361 +0000 UTC m=+3933.873351157" watchObservedRunningTime="2026-10-30 22:05:46.75266442 +0000 UTC m=+3933.873738166" Oct 30 22:05:47.720388 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:47.720348 4926 generic.go:356] "Generic (PLEG): container finished" podID="29140a1f-6928-4042-a96b-d74a0db8e138" containerID="52fbb9b72166ad418113d20bafc6381b41c0c67ce633f4ca283ba9a91eef7885" exitCode=0 Oct 30 22:05:47.720842 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:47.720432 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x" event={"ID":"29140a1f-6928-4042-a96b-d74a0db8e138","Type":"ContainerDied","Data":"52fbb9b72166ad418113d20bafc6381b41c0c67ce633f4ca283ba9a91eef7885"} Oct 30 22:05:48.860883 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:48.860855 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x" Oct 30 22:05:48.894405 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:48.894375 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29140a1f-6928-4042-a96b-d74a0db8e138-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29140a1f-6928-4042-a96b-d74a0db8e138-maas-api-xks-ca\") pod \"29140a1f-6928-4042-a96b-d74a0db8e138\" (UID: \"29140a1f-6928-4042-a96b-d74a0db8e138\") " Oct 30 22:05:48.894405 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:48.894418 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29140a1f-6928-4042-a96b-d74a0db8e138-kube-api-access-vb98b\" (UniqueName: \"kubernetes.io/projected/29140a1f-6928-4042-a96b-d74a0db8e138-kube-api-access-vb98b\") pod \"29140a1f-6928-4042-a96b-d74a0db8e138\" (UID: \"29140a1f-6928-4042-a96b-d74a0db8e138\") " Oct 30 22:05:48.894640 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:48.894498 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29140a1f-6928-4042-a96b-d74a0db8e138-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29140a1f-6928-4042-a96b-d74a0db8e138-maas-api-openshift-ca\") pod \"29140a1f-6928-4042-a96b-d74a0db8e138\" (UID: \"29140a1f-6928-4042-a96b-d74a0db8e138\") " Oct 30 22:05:48.894893 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:48.894867 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29140a1f-6928-4042-a96b-d74a0db8e138-maas-api-openshift-ca" pod "29140a1f-6928-4042-a96b-d74a0db8e138" (UID: "29140a1f-6928-4042-a96b-d74a0db8e138"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:48.896563 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:48.896541 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29140a1f-6928-4042-a96b-d74a0db8e138-maas-api-xks-ca" pod "29140a1f-6928-4042-a96b-d74a0db8e138" (UID: "29140a1f-6928-4042-a96b-d74a0db8e138"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:48.896656 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:48.896637 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29140a1f-6928-4042-a96b-d74a0db8e138-kube-api-access-vb98b" pod "29140a1f-6928-4042-a96b-d74a0db8e138" (UID: "29140a1f-6928-4042-a96b-d74a0db8e138"). InnerVolumeSpecName "kube-api-access-vb98b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:48.995603 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:48.995518 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29140a1f-6928-4042-a96b-d74a0db8e138-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:05:48.995603 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:48.995553 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb98b\" (UniqueName: \"kubernetes.io/projected/29140a1f-6928-4042-a96b-d74a0db8e138-kube-api-access-vb98b\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:05:48.995603 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:48.995564 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29140a1f-6928-4042-a96b-d74a0db8e138-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:05:49.734292 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:49.734248 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x" event={"ID":"29140a1f-6928-4042-a96b-d74a0db8e138","Type":"ContainerDied","Data":"0c6c2040f851ca4a4c014a947a169f8322a2ec6f99003b21e581c2c65fe99129"} Oct 30 22:05:49.734480 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:49.734288 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x" Oct 30 22:05:49.734480 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:49.734302 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c6c2040f851ca4a4c014a947a169f8322a2ec6f99003b21e581c2c65fe99129" Oct 30 22:05:58.811582 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:58.811538 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x"] Oct 30 22:05:58.821816 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:58.821785 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-d36cd3-wjt5x"] Oct 30 22:05:59.015156 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:05:59.015117 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29140a1f-6928-4042-a96b-d74a0db8e138" path="/var/lib/kubelet/pods/29140a1f-6928-4042-a96b-d74a0db8e138/volumes" Oct 30 22:06:00.634831 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:00.634779 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:06:07.489355 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:06:07.489315 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:06:24.861242 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:24.861201 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s"] Oct 30 22:06:24.861708 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:24.861458 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s" podUID="1f0cfe45-affa-48ba-aeea-1f2d5681a022" containerName="manager" containerID="cri-o://8693ec263bb3e9720e99db955db1f076c891ffcc3d075fe3df1a4ac45fb66f28" gracePeriod=10 Oct 30 22:06:25.120343 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:25.120275 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s" Oct 30 22:06:25.148601 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:25.148566 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f0cfe45-affa-48ba-aeea-1f2d5681a022-kube-api-access-tr8jw\" (UniqueName: \"kubernetes.io/projected/1f0cfe45-affa-48ba-aeea-1f2d5681a022-kube-api-access-tr8jw\") pod \"1f0cfe45-affa-48ba-aeea-1f2d5681a022\" (UID: \"1f0cfe45-affa-48ba-aeea-1f2d5681a022\") " Oct 30 22:06:25.148729 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:25.148716 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f0cfe45-affa-48ba-aeea-1f2d5681a022-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f0cfe45-affa-48ba-aeea-1f2d5681a022-extensions-socket-volume\") pod \"1f0cfe45-affa-48ba-aeea-1f2d5681a022\" (UID: \"1f0cfe45-affa-48ba-aeea-1f2d5681a022\") " Oct 30 22:06:25.149134 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:25.149104 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f0cfe45-affa-48ba-aeea-1f2d5681a022-extensions-socket-volume" pod "1f0cfe45-affa-48ba-aeea-1f2d5681a022" (UID: "1f0cfe45-affa-48ba-aeea-1f2d5681a022"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:25.150776 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:25.150752 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f0cfe45-affa-48ba-aeea-1f2d5681a022-kube-api-access-tr8jw" pod "1f0cfe45-affa-48ba-aeea-1f2d5681a022" (UID: "1f0cfe45-affa-48ba-aeea-1f2d5681a022"). InnerVolumeSpecName "kube-api-access-tr8jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:25.249708 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:25.249672 4926 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1f0cfe45-affa-48ba-aeea-1f2d5681a022-extensions-socket-volume\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:06:25.249708 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:25.249704 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr8jw\" (UniqueName: \"kubernetes.io/projected/1f0cfe45-affa-48ba-aeea-1f2d5681a022-kube-api-access-tr8jw\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:06:25.935494 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:25.935459 4926 generic.go:356] "Generic (PLEG): container finished" podID="1f0cfe45-affa-48ba-aeea-1f2d5681a022" containerID="8693ec263bb3e9720e99db955db1f076c891ffcc3d075fe3df1a4ac45fb66f28" exitCode=0 Oct 30 22:06:25.936067 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:25.935508 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s" Oct 30 22:06:25.936067 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:25.935540 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s" event={"ID":"1f0cfe45-affa-48ba-aeea-1f2d5681a022","Type":"ContainerDied","Data":"8693ec263bb3e9720e99db955db1f076c891ffcc3d075fe3df1a4ac45fb66f28"} Oct 30 22:06:25.936067 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:25.935605 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s" event={"ID":"1f0cfe45-affa-48ba-aeea-1f2d5681a022","Type":"ContainerDied","Data":"4a254854d50f0d278a95d7130c217a9fe89e0dba22ad27d813c3be025803211c"} Oct 30 22:06:25.936067 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:25.935637 4926 scope.go:122] "RemoveContainer" containerID="8693ec263bb3e9720e99db955db1f076c891ffcc3d075fe3df1a4ac45fb66f28" Oct 30 22:06:25.947531 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:25.947514 4926 scope.go:122] "RemoveContainer" containerID="17d48eb1b400e377c8f193cc7bef1b9c28f652b8f04c190ccf5282160320f2c7" Oct 30 22:06:25.955838 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:25.955819 4926 scope.go:122] "RemoveContainer" containerID="8693ec263bb3e9720e99db955db1f076c891ffcc3d075fe3df1a4ac45fb66f28" Oct 30 22:06:25.956125 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:06:25.956104 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8693ec263bb3e9720e99db955db1f076c891ffcc3d075fe3df1a4ac45fb66f28\": container with ID starting with 8693ec263bb3e9720e99db955db1f076c891ffcc3d075fe3df1a4ac45fb66f28 not found: ID does not exist" containerID="8693ec263bb3e9720e99db955db1f076c891ffcc3d075fe3df1a4ac45fb66f28" Oct 30 22:06:25.956222 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:25.956139 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8693ec263bb3e9720e99db955db1f076c891ffcc3d075fe3df1a4ac45fb66f28"} err="failed to get container status \"8693ec263bb3e9720e99db955db1f076c891ffcc3d075fe3df1a4ac45fb66f28\": rpc error: code = NotFound desc = could not find container \"8693ec263bb3e9720e99db955db1f076c891ffcc3d075fe3df1a4ac45fb66f28\": container with ID starting with 8693ec263bb3e9720e99db955db1f076c891ffcc3d075fe3df1a4ac45fb66f28 not found: ID does not exist" Oct 30 22:06:25.956222 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:25.956165 4926 scope.go:122] "RemoveContainer" containerID="17d48eb1b400e377c8f193cc7bef1b9c28f652b8f04c190ccf5282160320f2c7" Oct 30 22:06:25.956416 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:06:25.956399 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17d48eb1b400e377c8f193cc7bef1b9c28f652b8f04c190ccf5282160320f2c7\": container with ID starting with 17d48eb1b400e377c8f193cc7bef1b9c28f652b8f04c190ccf5282160320f2c7 not found: ID does not exist" containerID="17d48eb1b400e377c8f193cc7bef1b9c28f652b8f04c190ccf5282160320f2c7" Oct 30 22:06:25.956476 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:25.956425 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17d48eb1b400e377c8f193cc7bef1b9c28f652b8f04c190ccf5282160320f2c7"} err="failed to get container status \"17d48eb1b400e377c8f193cc7bef1b9c28f652b8f04c190ccf5282160320f2c7\": rpc error: code = NotFound desc = could not find container \"17d48eb1b400e377c8f193cc7bef1b9c28f652b8f04c190ccf5282160320f2c7\": container with ID starting with 17d48eb1b400e377c8f193cc7bef1b9c28f652b8f04c190ccf5282160320f2c7 not found: ID does not exist" Oct 30 22:06:25.994441 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:25.994416 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s"] Oct 30 22:06:26.005420 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:26.005392 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-jtg2s"] Oct 30 22:06:27.014160 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:06:27.014125 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f0cfe45-affa-48ba-aeea-1f2d5681a022" path="/var/lib/kubelet/pods/1f0cfe45-affa-48ba-aeea-1f2d5681a022/volumes" Oct 30 22:06:37.483308 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:06:37.483269 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:07:07.482754 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:07:07.482714 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:07:28.026137 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:28.026103 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngzg7"] Oct 30 22:07:28.026591 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:28.026524 4926 cpu_manager.go:414] "containerMap: removing container" podUID="29140a1f-6928-4042-a96b-d74a0db8e138" containerName="revoke-keys" Oct 30 22:07:28.026591 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:28.026535 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29140a1f-6928-4042-a96b-d74a0db8e138" containerName="revoke-keys" Oct 30 22:07:28.026591 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:28.026553 4926 cpu_manager.go:414] "containerMap: removing container" podUID="1f0cfe45-affa-48ba-aeea-1f2d5681a022" containerName="manager" Oct 30 22:07:28.026591 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:28.026558 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0cfe45-affa-48ba-aeea-1f2d5681a022" containerName="manager" Oct 30 22:07:28.026591 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:28.026577 4926 cpu_manager.go:414] "containerMap: removing container" podUID="1f0cfe45-affa-48ba-aeea-1f2d5681a022" containerName="manager" Oct 30 22:07:28.026591 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:28.026583 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f0cfe45-affa-48ba-aeea-1f2d5681a022" containerName="manager" Oct 30 22:07:28.026800 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:28.026642 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="29140a1f-6928-4042-a96b-d74a0db8e138" containerName="revoke-keys" Oct 30 22:07:28.026800 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:28.026653 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0cfe45-affa-48ba-aeea-1f2d5681a022" containerName="manager" Oct 30 22:07:28.029668 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:28.029648 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngzg7" Oct 30 22:07:28.048013 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:28.047990 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-rps5z\"" Oct 30 22:07:28.054621 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:28.054595 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngzg7"] Oct 30 22:07:28.152337 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:28.152300 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8fe515a-6a45-4d3c-9715-4a156d26a237-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-ngzg7\" (UID: \"e8fe515a-6a45-4d3c-9715-4a156d26a237\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngzg7" Oct 30 22:07:28.152524 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:28.152476 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxzk9\" (UniqueName: \"kubernetes.io/projected/e8fe515a-6a45-4d3c-9715-4a156d26a237-kube-api-access-xxzk9\") pod \"kuadrant-operator-controller-manager-678d87766f-ngzg7\" (UID: \"e8fe515a-6a45-4d3c-9715-4a156d26a237\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngzg7" Oct 30 22:07:28.253021 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:28.252974 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxzk9\" (UniqueName: \"kubernetes.io/projected/e8fe515a-6a45-4d3c-9715-4a156d26a237-kube-api-access-xxzk9\") pod \"kuadrant-operator-controller-manager-678d87766f-ngzg7\" (UID: \"e8fe515a-6a45-4d3c-9715-4a156d26a237\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngzg7" Oct 30 22:07:28.253021 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:28.253047 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8fe515a-6a45-4d3c-9715-4a156d26a237-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-ngzg7\" (UID: \"e8fe515a-6a45-4d3c-9715-4a156d26a237\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngzg7" Oct 30 22:07:28.253538 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:28.253515 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e8fe515a-6a45-4d3c-9715-4a156d26a237-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-ngzg7\" (UID: \"e8fe515a-6a45-4d3c-9715-4a156d26a237\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngzg7" Oct 30 22:07:28.274952 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:28.274917 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxzk9\" (UniqueName: \"kubernetes.io/projected/e8fe515a-6a45-4d3c-9715-4a156d26a237-kube-api-access-xxzk9\") pod \"kuadrant-operator-controller-manager-678d87766f-ngzg7\" (UID: \"e8fe515a-6a45-4d3c-9715-4a156d26a237\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngzg7" Oct 30 22:07:28.342970 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:28.342877 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngzg7" Oct 30 22:07:28.669276 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:28.669242 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngzg7"] Oct 30 22:07:28.673768 ip-10-0-142-3 kubenswrapper[4926]: W1030 22:07:28.673726 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fe515a_6a45_4d3c_9715_4a156d26a237.slice/crio-889b5cf90d33b064d8a2bdcbb7abd2e5724305a7d75c60cec250d68c13d395b2 WatchSource:0}: Error finding container 889b5cf90d33b064d8a2bdcbb7abd2e5724305a7d75c60cec250d68c13d395b2: Status 404 returned error can't find the container with id 889b5cf90d33b064d8a2bdcbb7abd2e5724305a7d75c60cec250d68c13d395b2 Oct 30 22:07:29.280813 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:29.280769 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngzg7" event={"ID":"e8fe515a-6a45-4d3c-9715-4a156d26a237","Type":"ContainerStarted","Data":"1760c8f7e3b050ab30d26f6aa9f4e065ee866a4d7b40b69b770457adbc2fe86c"} Oct 30 22:07:29.281265 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:29.280835 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngzg7" Oct 30 22:07:29.281265 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:29.280858 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngzg7" event={"ID":"e8fe515a-6a45-4d3c-9715-4a156d26a237","Type":"ContainerStarted","Data":"889b5cf90d33b064d8a2bdcbb7abd2e5724305a7d75c60cec250d68c13d395b2"} Oct 30 22:07:29.348224 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:29.348168 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngzg7" podStartSLOduration=1.348154812 podStartE2EDuration="1.348154812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:29.336217693 +0000 UTC m=+4036.457291473" watchObservedRunningTime="2026-10-30 22:07:29.348154812 +0000 UTC m=+4036.469228561" Oct 30 22:07:37.484002 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:07:37.483957 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:07:40.288654 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:40.288612 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-ngzg7" Oct 30 22:07:48.334169 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:07:48.334121 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:08:03.600819 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:03.600769 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:08:07.560060 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:08:07.560005 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:08:09.089068 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:09.088853 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:08:09.137239 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:09.137190 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-rh5rw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:08:09.137437 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:09.137265 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-rh5rw" podUID="ed448351-dac3-4d81-9e30-bce9475607eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:08:24.938925 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:24.938842 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:08:35.211989 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:35.211927 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:08:37.488597 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:08:37.488556 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:08:46.483686 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.483637 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk"] Oct 30 22:08:46.484442 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.484410 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f0cfe45-affa-48ba-aeea-1f2d5681a022" containerName="manager" Oct 30 22:08:46.487979 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.487945 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:46.507255 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.507215 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk"] Oct 30 22:08:46.513984 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.513929 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-9b97ce\"/\"kube-root-ca.crt\"" Oct 30 22:08:46.514235 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.514012 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-9b97ce\"/\"openshift-service-ca.crt\"" Oct 30 22:08:46.516471 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.516443 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-9b97ce\"/\"default-dockercfg-vrfq8\"" Oct 30 22:08:46.519863 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.519836 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-9b97ce\"/\"simulator-single-main-9b97ce-kserve-self-signed-certs\"" Oct 30 22:08:46.523098 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.523072 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feee8741-1619-4cd7-8930-d8c2a14f1678-tls-certs\") pod \"simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:46.523248 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.523114 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-dshm\") pod \"simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:46.523248 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.523150 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnm9w\" (UniqueName: \"kubernetes.io/projected/feee8741-1619-4cd7-8930-d8c2a14f1678-kube-api-access-fnm9w\") pod \"simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:46.523248 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.523201 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-home\") pod \"simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:46.523248 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.523236 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-tmp-dir\") pod \"simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:46.523466 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.523291 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-model-cache\") pod \"simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:46.624566 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.624515 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnm9w\" (UniqueName: \"kubernetes.io/projected/feee8741-1619-4cd7-8930-d8c2a14f1678-kube-api-access-fnm9w\") pod \"simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:46.624815 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.624612 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-home\") pod \"simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:46.624815 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.624654 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-tmp-dir\") pod \"simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:46.624815 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.624736 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-model-cache\") pod \"simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:46.624815 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.624775 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feee8741-1619-4cd7-8930-d8c2a14f1678-tls-certs\") pod \"simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:46.625094 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.624829 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-dshm\") pod \"simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:46.625448 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.625204 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-tmp-dir\") pod \"simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:46.625448 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.625243 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-home\") pod \"simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:46.625448 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.625321 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-model-cache\") pod \"simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:46.627587 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.627557 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-dshm\") pod \"simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:46.628066 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.628026 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feee8741-1619-4cd7-8930-d8c2a14f1678-tls-certs\") pod \"simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:46.645584 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.645554 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnm9w\" (UniqueName: \"kubernetes.io/projected/feee8741-1619-4cd7-8930-d8c2a14f1678-kube-api-access-fnm9w\") pod \"simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:46.805017 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:46.804916 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:47.105639 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:47.105609 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk"] Oct 30 22:08:47.106814 ip-10-0-142-3 kubenswrapper[4926]: W1030 22:08:47.106788 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfeee8741_1619_4cd7_8930_d8c2a14f1678.slice/crio-418ddcd5422e385c860f86c34a29edb249fd14a6775d1f3366a80f87adc436af WatchSource:0}: Error finding container 418ddcd5422e385c860f86c34a29edb249fd14a6775d1f3366a80f87adc436af: Status 404 returned error can't find the container with id 418ddcd5422e385c860f86c34a29edb249fd14a6775d1f3366a80f87adc436af Oct 30 22:08:47.785121 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:47.785070 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" event={"ID":"feee8741-1619-4cd7-8930-d8c2a14f1678","Type":"ContainerStarted","Data":"06ccb687ec03d97ebc49bcd31c63c1eb7fcc949cca10066bfbcd416ffea30bd1"} Oct 30 22:08:47.785121 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:47.785124 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" event={"ID":"feee8741-1619-4cd7-8930-d8c2a14f1678","Type":"ContainerStarted","Data":"418ddcd5422e385c860f86c34a29edb249fd14a6775d1f3366a80f87adc436af"} Oct 30 22:08:47.805059 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:47.805005 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:47.820799 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:47.820746 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" podStartSLOduration=1.82073479 podStartE2EDuration="1.82073479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:47.819778728 +0000 UTC m=+4114.940852474" watchObservedRunningTime="2026-10-30 22:08:47.82073479 +0000 UTC m=+4114.941808529" Oct 30 22:08:49.033116 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:49.033077 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:08:56.805468 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:56.805424 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:56.818245 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:56.818216 4926 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:08:56.852580 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:08:56.852545 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:09:05.046131 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:05.046090 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:09:07.483696 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:09:07.483641 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:09:17.067517 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:17.067474 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-579c8af2-fwnq4"] Oct 30 22:09:17.070683 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:17.070662 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-579c8af2-fwnq4" Oct 30 22:09:17.085126 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:17.085095 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:09:17.085364 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:17.085223 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 22:09:17.098703 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:17.098671 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-579c8af2-fwnq4"] Oct 30 22:09:17.120235 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:17.120204 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:09:17.120235 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:17.120213 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:09:17.228814 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:17.228768 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff696058-a744-4301-b5d5-2c5edfb4e5ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-579c8af2-fwnq4\" (UID: \"ff696058-a744-4301-b5d5-2c5edfb4e5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579c8af2-fwnq4" Oct 30 22:09:17.229092 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:17.228909 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff696058-a744-4301-b5d5-2c5edfb4e5ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-579c8af2-fwnq4\" (UID: \"ff696058-a744-4301-b5d5-2c5edfb4e5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579c8af2-fwnq4" Oct 30 22:09:17.229092 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:17.228951 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d96fc\" (UniqueName: \"kubernetes.io/projected/ff696058-a744-4301-b5d5-2c5edfb4e5ec-kube-api-access-d96fc\") pod \"maas-api-revoke-sub-579c8af2-fwnq4\" (UID: \"ff696058-a744-4301-b5d5-2c5edfb4e5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579c8af2-fwnq4" Oct 30 22:09:17.330407 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:17.330306 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff696058-a744-4301-b5d5-2c5edfb4e5ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-579c8af2-fwnq4\" (UID: \"ff696058-a744-4301-b5d5-2c5edfb4e5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579c8af2-fwnq4" Oct 30 22:09:17.330407 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:17.330349 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d96fc\" (UniqueName: \"kubernetes.io/projected/ff696058-a744-4301-b5d5-2c5edfb4e5ec-kube-api-access-d96fc\") pod \"maas-api-revoke-sub-579c8af2-fwnq4\" (UID: \"ff696058-a744-4301-b5d5-2c5edfb4e5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579c8af2-fwnq4" Oct 30 22:09:17.330682 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:17.330425 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff696058-a744-4301-b5d5-2c5edfb4e5ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-579c8af2-fwnq4\" (UID: \"ff696058-a744-4301-b5d5-2c5edfb4e5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579c8af2-fwnq4" Oct 30 22:09:17.331086 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:17.331061 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff696058-a744-4301-b5d5-2c5edfb4e5ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-579c8af2-fwnq4\" (UID: \"ff696058-a744-4301-b5d5-2c5edfb4e5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579c8af2-fwnq4" Oct 30 22:09:17.332813 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:17.332792 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff696058-a744-4301-b5d5-2c5edfb4e5ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-579c8af2-fwnq4\" (UID: \"ff696058-a744-4301-b5d5-2c5edfb4e5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579c8af2-fwnq4" Oct 30 22:09:17.352318 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:17.352288 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d96fc\" (UniqueName: \"kubernetes.io/projected/ff696058-a744-4301-b5d5-2c5edfb4e5ec-kube-api-access-d96fc\") pod \"maas-api-revoke-sub-579c8af2-fwnq4\" (UID: \"ff696058-a744-4301-b5d5-2c5edfb4e5ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-579c8af2-fwnq4" Oct 30 22:09:17.384810 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:17.384766 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-579c8af2-fwnq4" Oct 30 22:09:17.676890 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:17.676853 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-579c8af2-fwnq4"] Oct 30 22:09:17.678143 ip-10-0-142-3 kubenswrapper[4926]: W1030 22:09:17.678112 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff696058_a744_4301_b5d5_2c5edfb4e5ec.slice/crio-1397edaa7df696aa025b800ad264e8a92b3eb9a91bbe64cf1afe525440a32999 WatchSource:0}: Error finding container 1397edaa7df696aa025b800ad264e8a92b3eb9a91bbe64cf1afe525440a32999: Status 404 returned error can't find the container with id 1397edaa7df696aa025b800ad264e8a92b3eb9a91bbe64cf1afe525440a32999 Oct 30 22:09:17.966853 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:17.966750 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-579c8af2-fwnq4" event={"ID":"ff696058-a744-4301-b5d5-2c5edfb4e5ec","Type":"ContainerStarted","Data":"1397edaa7df696aa025b800ad264e8a92b3eb9a91bbe64cf1afe525440a32999"} Oct 30 22:09:18.981954 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:18.981852 4926 generic.go:356] "Generic (PLEG): container finished" podID="ff696058-a744-4301-b5d5-2c5edfb4e5ec" containerID="796dac2e2b5584de0e13aa4aa3e410578e0809d509f17dbf4dd004a478883236" exitCode=0 Oct 30 22:09:18.982385 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:18.981977 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-579c8af2-fwnq4" event={"ID":"ff696058-a744-4301-b5d5-2c5edfb4e5ec","Type":"ContainerDied","Data":"796dac2e2b5584de0e13aa4aa3e410578e0809d509f17dbf4dd004a478883236"} Oct 30 22:09:20.117970 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:20.117943 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-579c8af2-fwnq4" Oct 30 22:09:20.260840 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:20.260749 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff696058-a744-4301-b5d5-2c5edfb4e5ec-kube-api-access-d96fc\" (UniqueName: \"kubernetes.io/projected/ff696058-a744-4301-b5d5-2c5edfb4e5ec-kube-api-access-d96fc\") pod \"ff696058-a744-4301-b5d5-2c5edfb4e5ec\" (UID: \"ff696058-a744-4301-b5d5-2c5edfb4e5ec\") " Oct 30 22:09:20.260840 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:20.260804 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff696058-a744-4301-b5d5-2c5edfb4e5ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff696058-a744-4301-b5d5-2c5edfb4e5ec-maas-api-xks-ca\") pod \"ff696058-a744-4301-b5d5-2c5edfb4e5ec\" (UID: \"ff696058-a744-4301-b5d5-2c5edfb4e5ec\") " Oct 30 22:09:20.261077 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:20.260922 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff696058-a744-4301-b5d5-2c5edfb4e5ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff696058-a744-4301-b5d5-2c5edfb4e5ec-maas-api-openshift-ca\") pod \"ff696058-a744-4301-b5d5-2c5edfb4e5ec\" (UID: \"ff696058-a744-4301-b5d5-2c5edfb4e5ec\") " Oct 30 22:09:20.261297 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:20.261271 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff696058-a744-4301-b5d5-2c5edfb4e5ec-maas-api-openshift-ca" pod "ff696058-a744-4301-b5d5-2c5edfb4e5ec" (UID: "ff696058-a744-4301-b5d5-2c5edfb4e5ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:20.263123 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:20.263090 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff696058-a744-4301-b5d5-2c5edfb4e5ec-maas-api-xks-ca" pod "ff696058-a744-4301-b5d5-2c5edfb4e5ec" (UID: "ff696058-a744-4301-b5d5-2c5edfb4e5ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:20.263302 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:20.263277 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff696058-a744-4301-b5d5-2c5edfb4e5ec-kube-api-access-d96fc" pod "ff696058-a744-4301-b5d5-2c5edfb4e5ec" (UID: "ff696058-a744-4301-b5d5-2c5edfb4e5ec"). InnerVolumeSpecName "kube-api-access-d96fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:20.362640 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:20.362593 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d96fc\" (UniqueName: \"kubernetes.io/projected/ff696058-a744-4301-b5d5-2c5edfb4e5ec-kube-api-access-d96fc\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:09:20.362640 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:20.362631 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff696058-a744-4301-b5d5-2c5edfb4e5ec-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:09:20.362640 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:20.362642 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff696058-a744-4301-b5d5-2c5edfb4e5ec-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:09:20.995700 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:20.995670 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-579c8af2-fwnq4" Oct 30 22:09:20.995904 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:20.995703 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-579c8af2-fwnq4" event={"ID":"ff696058-a744-4301-b5d5-2c5edfb4e5ec","Type":"ContainerDied","Data":"1397edaa7df696aa025b800ad264e8a92b3eb9a91bbe64cf1afe525440a32999"} Oct 30 22:09:20.995904 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:20.995767 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1397edaa7df696aa025b800ad264e8a92b3eb9a91bbe64cf1afe525440a32999" Oct 30 22:09:28.107834 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:28.107796 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z"] Oct 30 22:09:28.108277 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:28.108259 4926 cpu_manager.go:414] "containerMap: removing container" podUID="ff696058-a744-4301-b5d5-2c5edfb4e5ec" containerName="revoke-keys" Oct 30 22:09:28.108277 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:28.108273 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff696058-a744-4301-b5d5-2c5edfb4e5ec" containerName="revoke-keys" Oct 30 22:09:28.108366 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:28.108352 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff696058-a744-4301-b5d5-2c5edfb4e5ec" containerName="revoke-keys" Oct 30 22:09:28.111554 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:28.111536 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z" Oct 30 22:09:28.124704 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:28.124683 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:09:28.132431 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:28.132401 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32edb97-702e-4922-b52c-8d21ded2f113-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z\" (UID: \"a32edb97-702e-4922-b52c-8d21ded2f113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z" Oct 30 22:09:28.132583 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:28.132439 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2vlb\" (UniqueName: \"kubernetes.io/projected/a32edb97-702e-4922-b52c-8d21ded2f113-kube-api-access-j2vlb\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z\" (UID: \"a32edb97-702e-4922-b52c-8d21ded2f113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z" Oct 30 22:09:28.132583 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:28.132461 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32edb97-702e-4922-b52c-8d21ded2f113-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z\" (UID: \"a32edb97-702e-4922-b52c-8d21ded2f113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z" Oct 30 22:09:28.135667 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:28.135636 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 22:09:28.135769 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:28.135695 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:09:28.135769 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:28.135730 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:09:28.138558 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:28.138534 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z"] Oct 30 22:09:28.233351 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:28.233315 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32edb97-702e-4922-b52c-8d21ded2f113-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z\" (UID: \"a32edb97-702e-4922-b52c-8d21ded2f113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z" Oct 30 22:09:28.233351 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:28.233356 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2vlb\" (UniqueName: \"kubernetes.io/projected/a32edb97-702e-4922-b52c-8d21ded2f113-kube-api-access-j2vlb\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z\" (UID: \"a32edb97-702e-4922-b52c-8d21ded2f113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z" Oct 30 22:09:28.233590 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:28.233374 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32edb97-702e-4922-b52c-8d21ded2f113-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z\" (UID: \"a32edb97-702e-4922-b52c-8d21ded2f113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z" Oct 30 22:09:28.234105 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:28.234079 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32edb97-702e-4922-b52c-8d21ded2f113-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z\" (UID: \"a32edb97-702e-4922-b52c-8d21ded2f113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z" Oct 30 22:09:28.235788 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:28.235762 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32edb97-702e-4922-b52c-8d21ded2f113-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z\" (UID: \"a32edb97-702e-4922-b52c-8d21ded2f113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z" Oct 30 22:09:28.250993 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:28.250968 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2vlb\" (UniqueName: \"kubernetes.io/projected/a32edb97-702e-4922-b52c-8d21ded2f113-kube-api-access-j2vlb\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z\" (UID: \"a32edb97-702e-4922-b52c-8d21ded2f113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z" Oct 30 22:09:28.424867 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:28.424828 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z" Oct 30 22:09:28.648454 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:28.648417 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:09:28.751266 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:28.747088 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z"] Oct 30 22:09:29.039448 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:29.039361 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z" event={"ID":"a32edb97-702e-4922-b52c-8d21ded2f113","Type":"ContainerStarted","Data":"fdaf2ea28c6685ed1ff6e4f0a6f77c86a2eea376a2474404ccf743b53d850db6"} Oct 30 22:09:30.047628 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:30.047543 4926 generic.go:356] "Generic (PLEG): container finished" podID="a32edb97-702e-4922-b52c-8d21ded2f113" containerID="92b54bc35fcf6b44a0e07ba94f9cc49cfbc6a2fc6dfc11af0f20037a2c51e06c" exitCode=0 Oct 30 22:09:30.047628 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:30.047594 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z" event={"ID":"a32edb97-702e-4922-b52c-8d21ded2f113","Type":"ContainerDied","Data":"92b54bc35fcf6b44a0e07ba94f9cc49cfbc6a2fc6dfc11af0f20037a2c51e06c"} Oct 30 22:09:31.203176 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:31.203148 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z" Oct 30 22:09:31.255897 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:31.255865 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a32edb97-702e-4922-b52c-8d21ded2f113-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32edb97-702e-4922-b52c-8d21ded2f113-maas-api-xks-ca\") pod \"a32edb97-702e-4922-b52c-8d21ded2f113\" (UID: \"a32edb97-702e-4922-b52c-8d21ded2f113\") " Oct 30 22:09:31.256084 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:31.255928 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a32edb97-702e-4922-b52c-8d21ded2f113-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32edb97-702e-4922-b52c-8d21ded2f113-maas-api-openshift-ca\") pod \"a32edb97-702e-4922-b52c-8d21ded2f113\" (UID: \"a32edb97-702e-4922-b52c-8d21ded2f113\") " Oct 30 22:09:31.256307 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:31.256284 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a32edb97-702e-4922-b52c-8d21ded2f113-maas-api-openshift-ca" pod "a32edb97-702e-4922-b52c-8d21ded2f113" (UID: "a32edb97-702e-4922-b52c-8d21ded2f113"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:31.258137 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:31.258112 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a32edb97-702e-4922-b52c-8d21ded2f113-maas-api-xks-ca" pod "a32edb97-702e-4922-b52c-8d21ded2f113" (UID: "a32edb97-702e-4922-b52c-8d21ded2f113"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:31.356552 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:31.356450 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a32edb97-702e-4922-b52c-8d21ded2f113-kube-api-access-j2vlb\" (UniqueName: \"kubernetes.io/projected/a32edb97-702e-4922-b52c-8d21ded2f113-kube-api-access-j2vlb\") pod \"a32edb97-702e-4922-b52c-8d21ded2f113\" (UID: \"a32edb97-702e-4922-b52c-8d21ded2f113\") " Oct 30 22:09:31.356739 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:31.356665 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a32edb97-702e-4922-b52c-8d21ded2f113-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:09:31.356739 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:31.356686 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a32edb97-702e-4922-b52c-8d21ded2f113-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:09:31.358691 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:31.358665 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a32edb97-702e-4922-b52c-8d21ded2f113-kube-api-access-j2vlb" pod "a32edb97-702e-4922-b52c-8d21ded2f113" (UID: "a32edb97-702e-4922-b52c-8d21ded2f113"). InnerVolumeSpecName "kube-api-access-j2vlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:31.457252 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:31.457203 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2vlb\" (UniqueName: \"kubernetes.io/projected/a32edb97-702e-4922-b52c-8d21ded2f113-kube-api-access-j2vlb\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:09:32.060267 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:32.060222 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z" event={"ID":"a32edb97-702e-4922-b52c-8d21ded2f113","Type":"ContainerDied","Data":"fdaf2ea28c6685ed1ff6e4f0a6f77c86a2eea376a2474404ccf743b53d850db6"} Oct 30 22:09:32.060472 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:32.060277 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdaf2ea28c6685ed1ff6e4f0a6f77c86a2eea376a2474404ccf743b53d850db6" Oct 30 22:09:32.060472 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:32.060298 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z" Oct 30 22:09:32.861480 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:32.861438 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk"] Oct 30 22:09:37.484386 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:09:37.484347 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:09:40.614752 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:40.614712 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z"] Oct 30 22:09:40.625462 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:40.625430 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b97ce-b4f6z"] Oct 30 22:09:41.014668 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:41.014631 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a32edb97-702e-4922-b52c-8d21ded2f113" path="/var/lib/kubelet/pods/a32edb97-702e-4922-b52c-8d21ded2f113/volumes" Oct 30 22:09:42.558972 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:42.558926 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:09:47.883675 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:47.883597 4926 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" podUID="feee8741-1619-4cd7-8930-d8c2a14f1678" containerName="main" containerID="cri-o://06ccb687ec03d97ebc49bcd31c63c1eb7fcc949cca10066bfbcd416ffea30bd1" gracePeriod=45 Oct 30 22:09:48.138972 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.138909 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:09:48.154556 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.154524 4926 generic.go:356] "Generic (PLEG): container finished" podID="feee8741-1619-4cd7-8930-d8c2a14f1678" containerID="06ccb687ec03d97ebc49bcd31c63c1eb7fcc949cca10066bfbcd416ffea30bd1" exitCode=0 Oct 30 22:09:48.154746 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.154568 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" Oct 30 22:09:48.154746 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.154610 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" event={"ID":"feee8741-1619-4cd7-8930-d8c2a14f1678","Type":"ContainerDied","Data":"06ccb687ec03d97ebc49bcd31c63c1eb7fcc949cca10066bfbcd416ffea30bd1"} Oct 30 22:09:48.154746 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.154661 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk" event={"ID":"feee8741-1619-4cd7-8930-d8c2a14f1678","Type":"ContainerDied","Data":"418ddcd5422e385c860f86c34a29edb249fd14a6775d1f3366a80f87adc436af"} Oct 30 22:09:48.154746 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.154687 4926 scope.go:122] "RemoveContainer" containerID="06ccb687ec03d97ebc49bcd31c63c1eb7fcc949cca10066bfbcd416ffea30bd1" Oct 30 22:09:48.163911 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.163894 4926 scope.go:122] "RemoveContainer" containerID="06ccb687ec03d97ebc49bcd31c63c1eb7fcc949cca10066bfbcd416ffea30bd1" Oct 30 22:09:48.164212 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:09:48.164188 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ccb687ec03d97ebc49bcd31c63c1eb7fcc949cca10066bfbcd416ffea30bd1\": container with ID starting with 06ccb687ec03d97ebc49bcd31c63c1eb7fcc949cca10066bfbcd416ffea30bd1 not found: ID does not exist" containerID="06ccb687ec03d97ebc49bcd31c63c1eb7fcc949cca10066bfbcd416ffea30bd1" Oct 30 22:09:48.164273 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.164222 4926 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ccb687ec03d97ebc49bcd31c63c1eb7fcc949cca10066bfbcd416ffea30bd1"} err="failed to get container status \"06ccb687ec03d97ebc49bcd31c63c1eb7fcc949cca10066bfbcd416ffea30bd1\": rpc error: code = NotFound desc = could not find container \"06ccb687ec03d97ebc49bcd31c63c1eb7fcc949cca10066bfbcd416ffea30bd1\": container with ID starting with 06ccb687ec03d97ebc49bcd31c63c1eb7fcc949cca10066bfbcd416ffea30bd1 not found: ID does not exist" Oct 30 22:09:48.240241 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.240203 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-tmp-dir\") pod \"feee8741-1619-4cd7-8930-d8c2a14f1678\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " Oct 30 22:09:48.240446 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.240273 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-dshm\" (UniqueName: \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-dshm\") pod \"feee8741-1619-4cd7-8930-d8c2a14f1678\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " Oct 30 22:09:48.240446 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.240315 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/feee8741-1619-4cd7-8930-d8c2a14f1678-tls-certs\" (UniqueName: \"kubernetes.io/secret/feee8741-1619-4cd7-8930-d8c2a14f1678-tls-certs\") pod \"feee8741-1619-4cd7-8930-d8c2a14f1678\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " Oct 30 22:09:48.240446 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.240339 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-home\" (UniqueName: \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-home\") pod \"feee8741-1619-4cd7-8930-d8c2a14f1678\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " Oct 30 22:09:48.240446 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.240364 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-model-cache\") pod \"feee8741-1619-4cd7-8930-d8c2a14f1678\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " Oct 30 22:09:48.240446 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.240382 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feee8741-1619-4cd7-8930-d8c2a14f1678-kube-api-access-fnm9w\" (UniqueName: \"kubernetes.io/projected/feee8741-1619-4cd7-8930-d8c2a14f1678-kube-api-access-fnm9w\") pod \"feee8741-1619-4cd7-8930-d8c2a14f1678\" (UID: \"feee8741-1619-4cd7-8930-d8c2a14f1678\") " Oct 30 22:09:48.240762 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.240523 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-tmp-dir" pod "feee8741-1619-4cd7-8930-d8c2a14f1678" (UID: "feee8741-1619-4cd7-8930-d8c2a14f1678"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:48.240762 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.240741 4926 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-tmp-dir\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:09:48.240889 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.240837 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-home" pod "feee8741-1619-4cd7-8930-d8c2a14f1678" (UID: "feee8741-1619-4cd7-8930-d8c2a14f1678"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:48.240889 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.240855 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-model-cache" pod "feee8741-1619-4cd7-8930-d8c2a14f1678" (UID: "feee8741-1619-4cd7-8930-d8c2a14f1678"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:48.242791 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.242748 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feee8741-1619-4cd7-8930-d8c2a14f1678-tls-certs" pod "feee8741-1619-4cd7-8930-d8c2a14f1678" (UID: "feee8741-1619-4cd7-8930-d8c2a14f1678"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:48.242967 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.242804 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-dshm" pod "feee8741-1619-4cd7-8930-d8c2a14f1678" (UID: "feee8741-1619-4cd7-8930-d8c2a14f1678"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:48.242967 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.242860 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feee8741-1619-4cd7-8930-d8c2a14f1678-kube-api-access-fnm9w" pod "feee8741-1619-4cd7-8930-d8c2a14f1678" (UID: "feee8741-1619-4cd7-8930-d8c2a14f1678"). InnerVolumeSpecName "kube-api-access-fnm9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:48.342122 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.342075 4926 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-dshm\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:09:48.342122 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.342114 4926 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feee8741-1619-4cd7-8930-d8c2a14f1678-tls-certs\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:09:48.342122 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.342127 4926 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-home\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:09:48.342388 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.342138 4926 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feee8741-1619-4cd7-8930-d8c2a14f1678-model-cache\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:09:48.342388 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.342152 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnm9w\" (UniqueName: \"kubernetes.io/projected/feee8741-1619-4cd7-8930-d8c2a14f1678-kube-api-access-fnm9w\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:09:48.497138 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.497097 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk"] Oct 30 22:09:48.504415 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:48.504383 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-9b97ce/simulator-single-main-9b97ce-kserve-86cbbccc9c-78tgk"] Oct 30 22:09:49.014987 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:49.014948 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feee8741-1619-4cd7-8930-d8c2a14f1678" path="/var/lib/kubelet/pods/feee8741-1619-4cd7-8930-d8c2a14f1678/volumes" Oct 30 22:09:56.149328 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:09:56.149244 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:10:07.484177 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:10:07.484121 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:10:10.920048 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:10:10.920003 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:10:14.078769 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:10:14.078738 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 22:10:14.090489 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:10:14.090462 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 22:10:26.621866 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:10:26.621822 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:10:37.025813 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:10:37.025772 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ebd54dd-wd6vb"] Oct 30 22:10:37.036423 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:10:37.036390 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ebd54dd-wd6vb"] Oct 30 22:10:37.482516 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:10:37.482471 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:10:38.816950 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:10:38.816913 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:10:39.015179 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:10:39.015143 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ef08bbd-125f-4fcf-a445-cd1fd966c3a0" path="/var/lib/kubelet/pods/1ef08bbd-125f-4fcf-a445-cd1fd966c3a0/volumes" Oct 30 22:10:46.058269 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:10:46.058223 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:10:53.816777 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:10:53.816735 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:10:55.877057 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:10:55.873153 4926 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-57f78bcd55-wxstq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:10:55.877057 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:10:55.873246 4926 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-57f78bcd55-wxstq" podUID="d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:11:01.156074 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:01.156020 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:11:07.483700 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:11:07.483658 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:11:17.316875 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:17.316835 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:11:35.889525 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:35.889419 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:11:37.484845 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:11:37.484808 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:11:44.895849 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:44.895799 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx"] Oct 30 22:11:44.896453 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:44.896428 4926 cpu_manager.go:414] "containerMap: removing container" podUID="feee8741-1619-4cd7-8930-d8c2a14f1678" containerName="main" Oct 30 22:11:44.896548 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:44.896455 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feee8741-1619-4cd7-8930-d8c2a14f1678" containerName="main" Oct 30 22:11:44.896548 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:44.896476 4926 cpu_manager.go:414] "containerMap: removing container" podUID="a32edb97-702e-4922-b52c-8d21ded2f113" containerName="revoke-keys" Oct 30 22:11:44.896548 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:44.896483 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a32edb97-702e-4922-b52c-8d21ded2f113" containerName="revoke-keys" Oct 30 22:11:44.896729 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:44.896601 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="feee8741-1619-4cd7-8930-d8c2a14f1678" containerName="main" Oct 30 22:11:44.896729 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:44.896615 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="a32edb97-702e-4922-b52c-8d21ded2f113" containerName="revoke-keys" Oct 30 22:11:44.900183 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:44.900159 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" Oct 30 22:11:44.926175 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:44.925916 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:11:44.926175 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:44.925937 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:11:44.926175 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:44.925943 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tnbrd\"" Oct 30 22:11:44.926175 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:44.925983 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:11:44.928498 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:44.928474 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx"] Oct 30 22:11:44.983605 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:44.983568 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx\" (UID: \"b51b6e85-a65f-4a74-9e19-d752f8c5e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" Oct 30 22:11:44.983778 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:44.983610 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bc56\" (UniqueName: \"kubernetes.io/projected/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-kube-api-access-2bc56\") pod \"maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx\" (UID: \"b51b6e85-a65f-4a74-9e19-d752f8c5e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" Oct 30 22:11:44.983778 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:44.983720 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx\" (UID: \"b51b6e85-a65f-4a74-9e19-d752f8c5e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" Oct 30 22:11:45.085077 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:45.085015 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx\" (UID: \"b51b6e85-a65f-4a74-9e19-d752f8c5e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" Oct 30 22:11:45.085077 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:45.085081 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bc56\" (UniqueName: \"kubernetes.io/projected/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-kube-api-access-2bc56\") pod \"maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx\" (UID: \"b51b6e85-a65f-4a74-9e19-d752f8c5e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" Oct 30 22:11:45.085360 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:45.085124 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx\" (UID: \"b51b6e85-a65f-4a74-9e19-d752f8c5e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" Oct 30 22:11:45.085785 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:45.085760 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx\" (UID: \"b51b6e85-a65f-4a74-9e19-d752f8c5e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" Oct 30 22:11:45.087449 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:45.087427 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx\" (UID: \"b51b6e85-a65f-4a74-9e19-d752f8c5e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" Oct 30 22:11:45.110570 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:11:45.110539 4926 projected.go:196] Error preparing data for projected volume kube-api-access-2bc56 for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 30 22:11:45.110727 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:11:45.110620 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-kube-api-access-2bc56 podName:b51b6e85-a65f-4a74-9e19-d752f8c5e6df nodeName:}" failed. No retries permitted until 2026-10-30 22:11:45.610599205 +0000 UTC m=+4292.731672943 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2bc56" (UniqueName: "kubernetes.io/projected/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-kube-api-access-2bc56") pod "maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" (UID: "b51b6e85-a65f-4a74-9e19-d752f8c5e6df") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 30 22:11:45.690759 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:45.690715 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bc56\" (UniqueName: \"kubernetes.io/projected/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-kube-api-access-2bc56\") pod \"maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx\" (UID: \"b51b6e85-a65f-4a74-9e19-d752f8c5e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" Oct 30 22:11:45.708174 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:11:45.708140 4926 projected.go:196] Error preparing data for projected volume kube-api-access-2bc56 for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx: failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 30 22:11:45.708360 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:11:45.708213 4926 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-kube-api-access-2bc56 podName:b51b6e85-a65f-4a74-9e19-d752f8c5e6df nodeName:}" failed. No retries permitted until 2026-10-30 22:11:46.708195767 +0000 UTC m=+4293.829269492 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bc56" (UniqueName: "kubernetes.io/projected/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-kube-api-access-2bc56") pod "maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" (UID: "b51b6e85-a65f-4a74-9e19-d752f8c5e6df") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found Oct 30 22:11:46.801447 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:46.801406 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bc56\" (UniqueName: \"kubernetes.io/projected/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-kube-api-access-2bc56\") pod \"maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx\" (UID: \"b51b6e85-a65f-4a74-9e19-d752f8c5e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" Oct 30 22:11:46.820160 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:46.820126 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bc56\" (UniqueName: \"kubernetes.io/projected/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-kube-api-access-2bc56\") pod \"maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx\" (UID: \"b51b6e85-a65f-4a74-9e19-d752f8c5e6df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" Oct 30 22:11:47.015003 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:47.014950 4926 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" secret="" err="secret \"maas-api-cleanup-dockercfg-tnbrd\" not found" Oct 30 22:11:47.015313 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:47.015282 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" Oct 30 22:11:47.317674 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:47.317640 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx"] Oct 30 22:11:47.319172 ip-10-0-142-3 kubenswrapper[4926]: W1030 22:11:47.319142 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51b6e85_a65f_4a74_9e19_d752f8c5e6df.slice/crio-c3fea0aa04c65c8792d63b433c9dfcd9e09c5d7b0f74a01118305fc292754300 WatchSource:0}: Error finding container c3fea0aa04c65c8792d63b433c9dfcd9e09c5d7b0f74a01118305fc292754300: Status 404 returned error can't find the container with id c3fea0aa04c65c8792d63b433c9dfcd9e09c5d7b0f74a01118305fc292754300 Oct 30 22:11:47.971932 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:47.971888 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" event={"ID":"b51b6e85-a65f-4a74-9e19-d752f8c5e6df","Type":"ContainerStarted","Data":"005165f41c0679ef2c1581d4067e26d13b15363da5325dbf9b7f3ee0c2a5e730"} Oct 30 22:11:47.972437 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:47.971944 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" event={"ID":"b51b6e85-a65f-4a74-9e19-d752f8c5e6df","Type":"ContainerStarted","Data":"c3fea0aa04c65c8792d63b433c9dfcd9e09c5d7b0f74a01118305fc292754300"} Oct 30 22:11:47.972437 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:47.972276 4926 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" secret="" err="secret \"maas-api-cleanup-dockercfg-tnbrd\" not found" Oct 30 22:11:48.010670 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:48.010611 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" podStartSLOduration=4.010598413 podStartE2EDuration="4.010598413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:48.008833541 +0000 UTC m=+4295.129907286" watchObservedRunningTime="2026-10-30 22:11:48.010598413 +0000 UTC m=+4295.131672159" Oct 30 22:11:48.979539 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:48.979500 4926 generic.go:356] "Generic (PLEG): container finished" podID="b51b6e85-a65f-4a74-9e19-d752f8c5e6df" containerID="005165f41c0679ef2c1581d4067e26d13b15363da5325dbf9b7f3ee0c2a5e730" exitCode=0 Oct 30 22:11:48.979962 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:48.979588 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" event={"ID":"b51b6e85-a65f-4a74-9e19-d752f8c5e6df","Type":"ContainerDied","Data":"005165f41c0679ef2c1581d4067e26d13b15363da5325dbf9b7f3ee0c2a5e730"} Oct 30 22:11:50.128171 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:50.128142 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" Oct 30 22:11:50.236811 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:50.236708 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-maas-api-openshift-ca\") pod \"b51b6e85-a65f-4a74-9e19-d752f8c5e6df\" (UID: \"b51b6e85-a65f-4a74-9e19-d752f8c5e6df\") " Oct 30 22:11:50.236811 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:50.236766 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-kube-api-access-2bc56\" (UniqueName: \"kubernetes.io/projected/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-kube-api-access-2bc56\") pod \"b51b6e85-a65f-4a74-9e19-d752f8c5e6df\" (UID: \"b51b6e85-a65f-4a74-9e19-d752f8c5e6df\") " Oct 30 22:11:50.237008 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:50.236827 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-maas-api-xks-ca\") pod \"b51b6e85-a65f-4a74-9e19-d752f8c5e6df\" (UID: \"b51b6e85-a65f-4a74-9e19-d752f8c5e6df\") " Oct 30 22:11:50.237233 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:50.237208 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-maas-api-openshift-ca" pod "b51b6e85-a65f-4a74-9e19-d752f8c5e6df" (UID: "b51b6e85-a65f-4a74-9e19-d752f8c5e6df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:11:50.239118 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:50.239084 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-kube-api-access-2bc56" pod "b51b6e85-a65f-4a74-9e19-d752f8c5e6df" (UID: "b51b6e85-a65f-4a74-9e19-d752f8c5e6df"). InnerVolumeSpecName "kube-api-access-2bc56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:11:50.239260 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:50.239140 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-maas-api-xks-ca" pod "b51b6e85-a65f-4a74-9e19-d752f8c5e6df" (UID: "b51b6e85-a65f-4a74-9e19-d752f8c5e6df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:11:50.338523 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:50.338478 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:11:50.338523 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:50.338516 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:11:50.338523 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:50.338527 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bc56\" (UniqueName: \"kubernetes.io/projected/b51b6e85-a65f-4a74-9e19-d752f8c5e6df-kube-api-access-2bc56\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:11:50.994968 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:50.994923 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" event={"ID":"b51b6e85-a65f-4a74-9e19-d752f8c5e6df","Type":"ContainerDied","Data":"c3fea0aa04c65c8792d63b433c9dfcd9e09c5d7b0f74a01118305fc292754300"} Oct 30 22:11:50.995230 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:50.994970 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx" Oct 30 22:11:50.995230 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:50.994992 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3fea0aa04c65c8792d63b433c9dfcd9e09c5d7b0f74a01118305fc292754300" Oct 30 22:11:57.300809 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:57.300768 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx"] Oct 30 22:11:57.311663 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:57.311628 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6ad3224d-bc4fx"] Oct 30 22:11:59.015346 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:59.015310 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b51b6e85-a65f-4a74-9e19-d752f8c5e6df" path="/var/lib/kubelet/pods/b51b6e85-a65f-4a74-9e19-d752f8c5e6df/volumes" Oct 30 22:11:59.050069 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:11:59.050017 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:12:07.484593 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:12:07.484550 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:12:12.343331 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:12:12.343264 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:12:20.206991 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:12:20.206955 4926 scope.go:122] "RemoveContainer" containerID="f53da5fb9b211421e1a982670266cb6a190fee901870b7d69f093b7522e8018b" Oct 30 22:12:20.216656 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:12:20.216628 4926 scope.go:122] "RemoveContainer" containerID="52fbb9b72166ad418113d20bafc6381b41c0c67ce633f4ca283ba9a91eef7885" Oct 30 22:12:37.483714 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:12:37.483667 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:13:07.482542 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:13:07.482501 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:13:37.483592 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:13:37.483552 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:13:42.227974 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:42.227936 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29"] Oct 30 22:13:42.228572 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:42.228551 4926 cpu_manager.go:414] "containerMap: removing container" podUID="b51b6e85-a65f-4a74-9e19-d752f8c5e6df" containerName="revoke-keys" Oct 30 22:13:42.228670 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:42.228574 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b51b6e85-a65f-4a74-9e19-d752f8c5e6df" containerName="revoke-keys" Oct 30 22:13:42.228762 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:42.228668 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="b51b6e85-a65f-4a74-9e19-d752f8c5e6df" containerName="revoke-keys" Oct 30 22:13:42.231920 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:42.231897 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29" Oct 30 22:13:42.257911 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:42.257877 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:13:42.258167 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:42.257888 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:13:42.258277 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:42.257911 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:13:42.258277 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:42.258017 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mssmb\"" Oct 30 22:13:42.260793 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:42.260766 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29"] Oct 30 22:13:42.330532 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:42.330493 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed8b3d30-8b65-485c-bd23-6bde4905c2a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29\" (UID: \"ed8b3d30-8b65-485c-bd23-6bde4905c2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29" Oct 30 22:13:42.330532 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:42.330535 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bsgv\" (UniqueName: \"kubernetes.io/projected/ed8b3d30-8b65-485c-bd23-6bde4905c2a8-kube-api-access-5bsgv\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29\" (UID: \"ed8b3d30-8b65-485c-bd23-6bde4905c2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29" Oct 30 22:13:42.330777 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:42.330750 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed8b3d30-8b65-485c-bd23-6bde4905c2a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29\" (UID: \"ed8b3d30-8b65-485c-bd23-6bde4905c2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29" Oct 30 22:13:42.431908 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:42.431870 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed8b3d30-8b65-485c-bd23-6bde4905c2a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29\" (UID: \"ed8b3d30-8b65-485c-bd23-6bde4905c2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29" Oct 30 22:13:42.432137 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:42.431912 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed8b3d30-8b65-485c-bd23-6bde4905c2a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29\" (UID: \"ed8b3d30-8b65-485c-bd23-6bde4905c2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29" Oct 30 22:13:42.432137 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:42.431941 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bsgv\" (UniqueName: \"kubernetes.io/projected/ed8b3d30-8b65-485c-bd23-6bde4905c2a8-kube-api-access-5bsgv\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29\" (UID: \"ed8b3d30-8b65-485c-bd23-6bde4905c2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29" Oct 30 22:13:42.432610 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:42.432586 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed8b3d30-8b65-485c-bd23-6bde4905c2a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29\" (UID: \"ed8b3d30-8b65-485c-bd23-6bde4905c2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29" Oct 30 22:13:42.434482 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:42.434458 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed8b3d30-8b65-485c-bd23-6bde4905c2a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29\" (UID: \"ed8b3d30-8b65-485c-bd23-6bde4905c2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29" Oct 30 22:13:42.449758 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:42.449731 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bsgv\" (UniqueName: \"kubernetes.io/projected/ed8b3d30-8b65-485c-bd23-6bde4905c2a8-kube-api-access-5bsgv\") pod \"maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29\" (UID: \"ed8b3d30-8b65-485c-bd23-6bde4905c2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29" Oct 30 22:13:42.545066 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:42.544955 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29" Oct 30 22:13:42.837181 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:42.837147 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29"] Oct 30 22:13:42.838192 ip-10-0-142-3 kubenswrapper[4926]: W1030 22:13:42.838167 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8b3d30_8b65_485c_bd23_6bde4905c2a8.slice/crio-9c7aadffcd77f39d82c2f9d6cc549100f5a3d9cc392a1f79c9dc65325e105b99 WatchSource:0}: Error finding container 9c7aadffcd77f39d82c2f9d6cc549100f5a3d9cc392a1f79c9dc65325e105b99: Status 404 returned error can't find the container with id 9c7aadffcd77f39d82c2f9d6cc549100f5a3d9cc392a1f79c9dc65325e105b99 Oct 30 22:13:43.640943 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:43.640904 4926 generic.go:356] "Generic (PLEG): container finished" podID="ed8b3d30-8b65-485c-bd23-6bde4905c2a8" containerID="7d033a995284d2f598897cd1c961bc16ff7f86034439c82c0cc8a7aafd5cea35" exitCode=0 Oct 30 22:13:43.641346 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:43.640997 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29" event={"ID":"ed8b3d30-8b65-485c-bd23-6bde4905c2a8","Type":"ContainerDied","Data":"7d033a995284d2f598897cd1c961bc16ff7f86034439c82c0cc8a7aafd5cea35"} Oct 30 22:13:43.641346 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:43.641070 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29" event={"ID":"ed8b3d30-8b65-485c-bd23-6bde4905c2a8","Type":"ContainerStarted","Data":"9c7aadffcd77f39d82c2f9d6cc549100f5a3d9cc392a1f79c9dc65325e105b99"} Oct 30 22:13:44.788368 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:44.788339 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29" Oct 30 22:13:44.853290 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:44.853253 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed8b3d30-8b65-485c-bd23-6bde4905c2a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed8b3d30-8b65-485c-bd23-6bde4905c2a8-maas-api-xks-ca\") pod \"ed8b3d30-8b65-485c-bd23-6bde4905c2a8\" (UID: \"ed8b3d30-8b65-485c-bd23-6bde4905c2a8\") " Oct 30 22:13:44.853483 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:44.853393 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed8b3d30-8b65-485c-bd23-6bde4905c2a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed8b3d30-8b65-485c-bd23-6bde4905c2a8-maas-api-openshift-ca\") pod \"ed8b3d30-8b65-485c-bd23-6bde4905c2a8\" (UID: \"ed8b3d30-8b65-485c-bd23-6bde4905c2a8\") " Oct 30 22:13:44.853483 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:44.853444 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed8b3d30-8b65-485c-bd23-6bde4905c2a8-kube-api-access-5bsgv\" (UniqueName: \"kubernetes.io/projected/ed8b3d30-8b65-485c-bd23-6bde4905c2a8-kube-api-access-5bsgv\") pod \"ed8b3d30-8b65-485c-bd23-6bde4905c2a8\" (UID: \"ed8b3d30-8b65-485c-bd23-6bde4905c2a8\") " Oct 30 22:13:44.853778 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:44.853755 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed8b3d30-8b65-485c-bd23-6bde4905c2a8-maas-api-openshift-ca" pod "ed8b3d30-8b65-485c-bd23-6bde4905c2a8" (UID: "ed8b3d30-8b65-485c-bd23-6bde4905c2a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:13:44.855512 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:44.855479 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed8b3d30-8b65-485c-bd23-6bde4905c2a8-maas-api-xks-ca" pod "ed8b3d30-8b65-485c-bd23-6bde4905c2a8" (UID: "ed8b3d30-8b65-485c-bd23-6bde4905c2a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:13:44.855605 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:44.855582 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8b3d30-8b65-485c-bd23-6bde4905c2a8-kube-api-access-5bsgv" pod "ed8b3d30-8b65-485c-bd23-6bde4905c2a8" (UID: "ed8b3d30-8b65-485c-bd23-6bde4905c2a8"). InnerVolumeSpecName "kube-api-access-5bsgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:13:44.954631 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:44.954586 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed8b3d30-8b65-485c-bd23-6bde4905c2a8-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:13:44.954631 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:44.954624 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bsgv\" (UniqueName: \"kubernetes.io/projected/ed8b3d30-8b65-485c-bd23-6bde4905c2a8-kube-api-access-5bsgv\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:13:44.954631 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:44.954634 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed8b3d30-8b65-485c-bd23-6bde4905c2a8-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:13:45.662526 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:45.662495 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29" Oct 30 22:13:45.662714 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:45.662528 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29" event={"ID":"ed8b3d30-8b65-485c-bd23-6bde4905c2a8","Type":"ContainerDied","Data":"9c7aadffcd77f39d82c2f9d6cc549100f5a3d9cc392a1f79c9dc65325e105b99"} Oct 30 22:13:45.662714 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:45.662581 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c7aadffcd77f39d82c2f9d6cc549100f5a3d9cc392a1f79c9dc65325e105b99" Oct 30 22:13:59.082359 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:59.082316 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29"] Oct 30 22:13:59.093147 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:13:59.093114 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-8c3a04-xcv29"] Oct 30 22:14:00.900854 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:00.900802 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:14:01.015254 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:01.015221 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed8b3d30-8b65-485c-bd23-6bde4905c2a8" path="/var/lib/kubelet/pods/ed8b3d30-8b65-485c-bd23-6bde4905c2a8/volumes" Oct 30 22:14:07.484961 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:14:07.484855 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:14:08.629915 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:08.623364 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:14:16.841688 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:16.841649 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:14:21.030222 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:21.030186 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-579c8af2-fwnq4"] Oct 30 22:14:21.040774 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:21.040720 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-579c8af2-fwnq4"] Oct 30 22:14:23.014068 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:23.014017 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff696058-a744-4301-b5d5-2c5edfb4e5ec" path="/var/lib/kubelet/pods/ff696058-a744-4301-b5d5-2c5edfb4e5ec/volumes" Oct 30 22:14:23.333335 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:23.333243 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v7k28"] Oct 30 22:14:23.333777 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:23.333757 4926 cpu_manager.go:414] "containerMap: removing container" podUID="ed8b3d30-8b65-485c-bd23-6bde4905c2a8" containerName="revoke-keys" Oct 30 22:14:23.333885 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:23.333779 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed8b3d30-8b65-485c-bd23-6bde4905c2a8" containerName="revoke-keys" Oct 30 22:14:23.333945 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:23.333921 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed8b3d30-8b65-485c-bd23-6bde4905c2a8" containerName="revoke-keys" Oct 30 22:14:23.338303 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:23.338282 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v7k28" Oct 30 22:14:23.354276 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:23.354250 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v7k28"] Oct 30 22:14:23.362106 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:23.362079 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:14:23.362106 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:23.362104 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:14:23.362279 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:23.362092 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:14:23.362279 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:23.362182 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mssmb\"" Oct 30 22:14:23.383426 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:23.383392 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrcpw\" (UniqueName: \"kubernetes.io/projected/3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60-kube-api-access-nrcpw\") pod \"maas-api-revoke-sub-f03629db-v7k28\" (UID: \"3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v7k28" Oct 30 22:14:23.383563 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:23.383448 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-v7k28\" (UID: \"3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v7k28" Oct 30 22:14:23.383563 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:23.383515 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-v7k28\" (UID: \"3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v7k28" Oct 30 22:14:23.485006 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:23.484971 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrcpw\" (UniqueName: \"kubernetes.io/projected/3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60-kube-api-access-nrcpw\") pod \"maas-api-revoke-sub-f03629db-v7k28\" (UID: \"3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v7k28" Oct 30 22:14:23.485226 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:23.485016 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-v7k28\" (UID: \"3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v7k28" Oct 30 22:14:23.485226 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:23.485051 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-v7k28\" (UID: \"3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v7k28" Oct 30 22:14:23.485911 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:23.485875 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-v7k28\" (UID: \"3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v7k28" Oct 30 22:14:23.487632 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:23.487606 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-v7k28\" (UID: \"3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v7k28" Oct 30 22:14:23.502622 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:23.502591 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrcpw\" (UniqueName: \"kubernetes.io/projected/3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60-kube-api-access-nrcpw\") pod \"maas-api-revoke-sub-f03629db-v7k28\" (UID: \"3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v7k28" Oct 30 22:14:23.651806 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:23.651701 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v7k28" Oct 30 22:14:23.945782 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:23.945753 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v7k28"] Oct 30 22:14:23.946853 ip-10-0-142-3 kubenswrapper[4926]: W1030 22:14:23.946828 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5611dd_1dcb_4d4d_b5e1_7ba153c4cd60.slice/crio-ce1dad960747467b29d887f114d272e205d7438637472c75c9bd7529ae1cac54 WatchSource:0}: Error finding container ce1dad960747467b29d887f114d272e205d7438637472c75c9bd7529ae1cac54: Status 404 returned error can't find the container with id ce1dad960747467b29d887f114d272e205d7438637472c75c9bd7529ae1cac54 Oct 30 22:14:24.920446 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:24.920404 4926 generic.go:356] "Generic (PLEG): container finished" podID="3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60" containerID="e60616972bef7d9156f7b46f04d0c323ee2141e23b8e9d1b9f966c5ca3b62f67" exitCode=0 Oct 30 22:14:24.920842 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:24.920484 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v7k28" event={"ID":"3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60","Type":"ContainerDied","Data":"e60616972bef7d9156f7b46f04d0c323ee2141e23b8e9d1b9f966c5ca3b62f67"} Oct 30 22:14:24.920842 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:24.920541 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v7k28" event={"ID":"3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60","Type":"ContainerStarted","Data":"ce1dad960747467b29d887f114d272e205d7438637472c75c9bd7529ae1cac54"} Oct 30 22:14:26.055012 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:26.054982 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v7k28" Oct 30 22:14:26.110871 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:26.110841 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60-maas-api-xks-ca\") pod \"3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60\" (UID: \"3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60\") " Oct 30 22:14:26.111087 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:26.110890 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60-maas-api-openshift-ca\") pod \"3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60\" (UID: \"3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60\") " Oct 30 22:14:26.111087 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:26.110952 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60-kube-api-access-nrcpw\" (UniqueName: \"kubernetes.io/projected/3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60-kube-api-access-nrcpw\") pod \"3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60\" (UID: \"3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60\") " Oct 30 22:14:26.111371 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:26.111337 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60-maas-api-openshift-ca" pod "3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60" (UID: "3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:26.113652 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:26.113625 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60-kube-api-access-nrcpw" pod "3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60" (UID: "3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60"). InnerVolumeSpecName "kube-api-access-nrcpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:26.113652 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:26.113626 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60-maas-api-xks-ca" pod "3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60" (UID: "3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:26.212901 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:26.212811 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:14:26.212901 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:26.212845 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:14:26.212901 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:26.212856 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrcpw\" (UniqueName: \"kubernetes.io/projected/3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60-kube-api-access-nrcpw\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:14:26.932923 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:26.932889 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v7k28" Oct 30 22:14:26.933148 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:26.932922 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-v7k28" event={"ID":"3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60","Type":"ContainerDied","Data":"ce1dad960747467b29d887f114d272e205d7438637472c75c9bd7529ae1cac54"} Oct 30 22:14:26.933148 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:26.932971 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce1dad960747467b29d887f114d272e205d7438637472c75c9bd7529ae1cac54" Oct 30 22:14:34.747084 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:34.747045 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:14:37.483500 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:14:37.483461 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:14:40.604928 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:40.604881 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-njcsj"] Oct 30 22:14:40.605590 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:40.605400 4926 cpu_manager.go:414] "containerMap: removing container" podUID="3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60" containerName="revoke-keys" Oct 30 22:14:40.605590 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:40.605420 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60" containerName="revoke-keys" Oct 30 22:14:40.605590 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:40.605548 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60" containerName="revoke-keys" Oct 30 22:14:40.609078 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:40.609056 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-njcsj" Oct 30 22:14:40.626590 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:40.626554 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-njcsj"] Oct 30 22:14:40.633378 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:40.633335 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:14:40.633378 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:40.633357 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:14:40.633580 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:40.633362 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:14:40.633580 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:40.633423 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mssmb\"" Oct 30 22:14:40.667762 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:40.667725 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0d1f7b-d853-4fad-a3bf-3075b28c8833-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-njcsj\" (UID: \"db0d1f7b-d853-4fad-a3bf-3075b28c8833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-njcsj" Oct 30 22:14:40.667940 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:40.667785 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0d1f7b-d853-4fad-a3bf-3075b28c8833-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-njcsj\" (UID: \"db0d1f7b-d853-4fad-a3bf-3075b28c8833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-njcsj" Oct 30 22:14:40.667940 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:40.667834 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnfch\" (UniqueName: \"kubernetes.io/projected/db0d1f7b-d853-4fad-a3bf-3075b28c8833-kube-api-access-pnfch\") pod \"maas-api-revoke-sub-0165d7d2-njcsj\" (UID: \"db0d1f7b-d853-4fad-a3bf-3075b28c8833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-njcsj" Oct 30 22:14:40.769192 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:40.769148 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0d1f7b-d853-4fad-a3bf-3075b28c8833-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-njcsj\" (UID: \"db0d1f7b-d853-4fad-a3bf-3075b28c8833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-njcsj" Oct 30 22:14:40.769394 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:40.769201 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0d1f7b-d853-4fad-a3bf-3075b28c8833-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-njcsj\" (UID: \"db0d1f7b-d853-4fad-a3bf-3075b28c8833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-njcsj" Oct 30 22:14:40.769394 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:40.769220 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnfch\" (UniqueName: \"kubernetes.io/projected/db0d1f7b-d853-4fad-a3bf-3075b28c8833-kube-api-access-pnfch\") pod \"maas-api-revoke-sub-0165d7d2-njcsj\" (UID: \"db0d1f7b-d853-4fad-a3bf-3075b28c8833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-njcsj" Oct 30 22:14:40.769834 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:40.769812 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0d1f7b-d853-4fad-a3bf-3075b28c8833-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-njcsj\" (UID: \"db0d1f7b-d853-4fad-a3bf-3075b28c8833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-njcsj" Oct 30 22:14:40.771701 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:40.771676 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0d1f7b-d853-4fad-a3bf-3075b28c8833-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-njcsj\" (UID: \"db0d1f7b-d853-4fad-a3bf-3075b28c8833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-njcsj" Oct 30 22:14:40.788143 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:40.788114 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnfch\" (UniqueName: \"kubernetes.io/projected/db0d1f7b-d853-4fad-a3bf-3075b28c8833-kube-api-access-pnfch\") pod \"maas-api-revoke-sub-0165d7d2-njcsj\" (UID: \"db0d1f7b-d853-4fad-a3bf-3075b28c8833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-njcsj" Oct 30 22:14:40.921693 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:40.921661 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-njcsj" Oct 30 22:14:41.218198 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:41.218138 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-njcsj"] Oct 30 22:14:41.221376 ip-10-0-142-3 kubenswrapper[4926]: W1030 22:14:41.221344 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0d1f7b_d853_4fad_a3bf_3075b28c8833.slice/crio-0afbc6fbc7b9afdd80266c8857f6e84580e954f8a860ecee1091854edf61af8c WatchSource:0}: Error finding container 0afbc6fbc7b9afdd80266c8857f6e84580e954f8a860ecee1091854edf61af8c: Status 404 returned error can't find the container with id 0afbc6fbc7b9afdd80266c8857f6e84580e954f8a860ecee1091854edf61af8c Oct 30 22:14:42.061677 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:42.061642 4926 generic.go:356] "Generic (PLEG): container finished" podID="db0d1f7b-d853-4fad-a3bf-3075b28c8833" containerID="8f357b4fdf2e7d499db5ac0cbe49fd2742112a3de63a2569d4edd12bb9be332f" exitCode=0 Oct 30 22:14:42.062094 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:42.061716 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-njcsj" event={"ID":"db0d1f7b-d853-4fad-a3bf-3075b28c8833","Type":"ContainerDied","Data":"8f357b4fdf2e7d499db5ac0cbe49fd2742112a3de63a2569d4edd12bb9be332f"} Oct 30 22:14:42.062094 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:42.061767 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-njcsj" event={"ID":"db0d1f7b-d853-4fad-a3bf-3075b28c8833","Type":"ContainerStarted","Data":"0afbc6fbc7b9afdd80266c8857f6e84580e954f8a860ecee1091854edf61af8c"} Oct 30 22:14:43.205127 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:43.205097 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-njcsj" Oct 30 22:14:43.293264 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:43.293220 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0d1f7b-d853-4fad-a3bf-3075b28c8833-kube-api-access-pnfch\" (UniqueName: \"kubernetes.io/projected/db0d1f7b-d853-4fad-a3bf-3075b28c8833-kube-api-access-pnfch\") pod \"db0d1f7b-d853-4fad-a3bf-3075b28c8833\" (UID: \"db0d1f7b-d853-4fad-a3bf-3075b28c8833\") " Oct 30 22:14:43.293442 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:43.293328 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0d1f7b-d853-4fad-a3bf-3075b28c8833-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0d1f7b-d853-4fad-a3bf-3075b28c8833-maas-api-openshift-ca\") pod \"db0d1f7b-d853-4fad-a3bf-3075b28c8833\" (UID: \"db0d1f7b-d853-4fad-a3bf-3075b28c8833\") " Oct 30 22:14:43.293442 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:43.293351 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db0d1f7b-d853-4fad-a3bf-3075b28c8833-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0d1f7b-d853-4fad-a3bf-3075b28c8833-maas-api-xks-ca\") pod \"db0d1f7b-d853-4fad-a3bf-3075b28c8833\" (UID: \"db0d1f7b-d853-4fad-a3bf-3075b28c8833\") " Oct 30 22:14:43.293700 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:43.293663 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0d1f7b-d853-4fad-a3bf-3075b28c8833-maas-api-openshift-ca" pod "db0d1f7b-d853-4fad-a3bf-3075b28c8833" (UID: "db0d1f7b-d853-4fad-a3bf-3075b28c8833"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:43.295489 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:43.295459 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db0d1f7b-d853-4fad-a3bf-3075b28c8833-maas-api-xks-ca" pod "db0d1f7b-d853-4fad-a3bf-3075b28c8833" (UID: "db0d1f7b-d853-4fad-a3bf-3075b28c8833"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:43.295572 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:43.295463 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0d1f7b-d853-4fad-a3bf-3075b28c8833-kube-api-access-pnfch" pod "db0d1f7b-d853-4fad-a3bf-3075b28c8833" (UID: "db0d1f7b-d853-4fad-a3bf-3075b28c8833"). InnerVolumeSpecName "kube-api-access-pnfch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:43.395089 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:43.394969 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnfch\" (UniqueName: \"kubernetes.io/projected/db0d1f7b-d853-4fad-a3bf-3075b28c8833-kube-api-access-pnfch\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:14:43.395089 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:43.395002 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db0d1f7b-d853-4fad-a3bf-3075b28c8833-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:14:43.395089 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:43.395013 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db0d1f7b-d853-4fad-a3bf-3075b28c8833-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:14:44.074363 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:44.074326 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-njcsj" event={"ID":"db0d1f7b-d853-4fad-a3bf-3075b28c8833","Type":"ContainerDied","Data":"0afbc6fbc7b9afdd80266c8857f6e84580e954f8a860ecee1091854edf61af8c"} Oct 30 22:14:44.074555 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:44.074376 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0afbc6fbc7b9afdd80266c8857f6e84580e954f8a860ecee1091854edf61af8c" Oct 30 22:14:44.074555 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:44.074339 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-njcsj" Oct 30 22:14:51.512044 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:51.511992 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p5f66"] Oct 30 22:14:51.512560 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:51.512425 4926 cpu_manager.go:414] "containerMap: removing container" podUID="db0d1f7b-d853-4fad-a3bf-3075b28c8833" containerName="revoke-keys" Oct 30 22:14:51.512560 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:51.512438 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0d1f7b-d853-4fad-a3bf-3075b28c8833" containerName="revoke-keys" Oct 30 22:14:51.512560 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:51.512519 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0d1f7b-d853-4fad-a3bf-3075b28c8833" containerName="revoke-keys" Oct 30 22:14:51.515726 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:51.515707 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p5f66" Oct 30 22:14:51.531217 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:51.531187 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:14:51.541843 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:51.541808 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 22:14:51.542075 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:51.541874 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:14:51.542075 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:51.541930 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mssmb\"" Oct 30 22:14:51.544425 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:51.544400 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p5f66"] Oct 30 22:14:51.670570 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:51.670524 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2bce597-c86b-4665-88af-ac36651a05b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-p5f66\" (UID: \"e2bce597-c86b-4665-88af-ac36651a05b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p5f66" Oct 30 22:14:51.670760 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:51.670579 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2bce597-c86b-4665-88af-ac36651a05b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-p5f66\" (UID: \"e2bce597-c86b-4665-88af-ac36651a05b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p5f66" Oct 30 22:14:51.670760 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:51.670659 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr2j8\" (UniqueName: \"kubernetes.io/projected/e2bce597-c86b-4665-88af-ac36651a05b1-kube-api-access-hr2j8\") pod \"maas-api-revoke-sub-6b19338f-p5f66\" (UID: \"e2bce597-c86b-4665-88af-ac36651a05b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p5f66" Oct 30 22:14:51.772502 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:51.772409 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2bce597-c86b-4665-88af-ac36651a05b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-p5f66\" (UID: \"e2bce597-c86b-4665-88af-ac36651a05b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p5f66" Oct 30 22:14:51.772502 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:51.772462 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2bce597-c86b-4665-88af-ac36651a05b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-p5f66\" (UID: \"e2bce597-c86b-4665-88af-ac36651a05b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p5f66" Oct 30 22:14:51.772502 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:51.772490 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr2j8\" (UniqueName: \"kubernetes.io/projected/e2bce597-c86b-4665-88af-ac36651a05b1-kube-api-access-hr2j8\") pod \"maas-api-revoke-sub-6b19338f-p5f66\" (UID: \"e2bce597-c86b-4665-88af-ac36651a05b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p5f66" Oct 30 22:14:51.773191 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:51.773155 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2bce597-c86b-4665-88af-ac36651a05b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-p5f66\" (UID: \"e2bce597-c86b-4665-88af-ac36651a05b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p5f66" Oct 30 22:14:51.774959 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:51.774935 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2bce597-c86b-4665-88af-ac36651a05b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-p5f66\" (UID: \"e2bce597-c86b-4665-88af-ac36651a05b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p5f66" Oct 30 22:14:51.791576 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:51.791544 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr2j8\" (UniqueName: \"kubernetes.io/projected/e2bce597-c86b-4665-88af-ac36651a05b1-kube-api-access-hr2j8\") pod \"maas-api-revoke-sub-6b19338f-p5f66\" (UID: \"e2bce597-c86b-4665-88af-ac36651a05b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p5f66" Oct 30 22:14:51.829431 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:51.829400 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p5f66" Oct 30 22:14:52.123041 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:52.122952 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p5f66"] Oct 30 22:14:52.126323 ip-10-0-142-3 kubenswrapper[4926]: W1030 22:14:52.126291 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2bce597_c86b_4665_88af_ac36651a05b1.slice/crio-401d742ec481c182057f3e234f4a30125ca0d1ab86883c82d86ff02481a68b9a WatchSource:0}: Error finding container 401d742ec481c182057f3e234f4a30125ca0d1ab86883c82d86ff02481a68b9a: Status 404 returned error can't find the container with id 401d742ec481c182057f3e234f4a30125ca0d1ab86883c82d86ff02481a68b9a Oct 30 22:14:53.124001 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:53.123959 4926 generic.go:356] "Generic (PLEG): container finished" podID="e2bce597-c86b-4665-88af-ac36651a05b1" containerID="d504f0eaf0f68f080531b8879df8526fdb191e11c236050e6b01404f4d857df8" exitCode=0 Oct 30 22:14:53.124451 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:53.124023 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p5f66" event={"ID":"e2bce597-c86b-4665-88af-ac36651a05b1","Type":"ContainerDied","Data":"d504f0eaf0f68f080531b8879df8526fdb191e11c236050e6b01404f4d857df8"} Oct 30 22:14:53.124451 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:53.124099 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p5f66" event={"ID":"e2bce597-c86b-4665-88af-ac36651a05b1","Type":"ContainerStarted","Data":"401d742ec481c182057f3e234f4a30125ca0d1ab86883c82d86ff02481a68b9a"} Oct 30 22:14:54.264478 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:54.264455 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p5f66" Oct 30 22:14:54.401534 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:54.401439 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2bce597-c86b-4665-88af-ac36651a05b1-kube-api-access-hr2j8\" (UniqueName: \"kubernetes.io/projected/e2bce597-c86b-4665-88af-ac36651a05b1-kube-api-access-hr2j8\") pod \"e2bce597-c86b-4665-88af-ac36651a05b1\" (UID: \"e2bce597-c86b-4665-88af-ac36651a05b1\") " Oct 30 22:14:54.401534 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:54.401513 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2bce597-c86b-4665-88af-ac36651a05b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2bce597-c86b-4665-88af-ac36651a05b1-maas-api-xks-ca\") pod \"e2bce597-c86b-4665-88af-ac36651a05b1\" (UID: \"e2bce597-c86b-4665-88af-ac36651a05b1\") " Oct 30 22:14:54.401784 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:54.401578 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2bce597-c86b-4665-88af-ac36651a05b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2bce597-c86b-4665-88af-ac36651a05b1-maas-api-openshift-ca\") pod \"e2bce597-c86b-4665-88af-ac36651a05b1\" (UID: \"e2bce597-c86b-4665-88af-ac36651a05b1\") " Oct 30 22:14:54.401971 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:54.401945 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2bce597-c86b-4665-88af-ac36651a05b1-maas-api-openshift-ca" pod "e2bce597-c86b-4665-88af-ac36651a05b1" (UID: "e2bce597-c86b-4665-88af-ac36651a05b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:14:54.403661 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:54.403633 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2bce597-c86b-4665-88af-ac36651a05b1-maas-api-xks-ca" pod "e2bce597-c86b-4665-88af-ac36651a05b1" (UID: "e2bce597-c86b-4665-88af-ac36651a05b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:14:54.403780 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:54.403729 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2bce597-c86b-4665-88af-ac36651a05b1-kube-api-access-hr2j8" pod "e2bce597-c86b-4665-88af-ac36651a05b1" (UID: "e2bce597-c86b-4665-88af-ac36651a05b1"). InnerVolumeSpecName "kube-api-access-hr2j8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:14:54.502948 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:54.502912 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr2j8\" (UniqueName: \"kubernetes.io/projected/e2bce597-c86b-4665-88af-ac36651a05b1-kube-api-access-hr2j8\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:14:54.502948 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:54.502942 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2bce597-c86b-4665-88af-ac36651a05b1-maas-api-xks-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:14:54.502948 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:54.502953 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2bce597-c86b-4665-88af-ac36651a05b1-maas-api-openshift-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:14:55.136532 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:55.136497 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p5f66" Oct 30 22:14:55.136732 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:55.136540 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-p5f66" event={"ID":"e2bce597-c86b-4665-88af-ac36651a05b1","Type":"ContainerDied","Data":"401d742ec481c182057f3e234f4a30125ca0d1ab86883c82d86ff02481a68b9a"} Oct 30 22:14:55.136732 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:14:55.136589 4926 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="401d742ec481c182057f3e234f4a30125ca0d1ab86883c82d86ff02481a68b9a" Oct 30 22:15:00.231890 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:00.231847 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg"] Oct 30 22:15:00.232505 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:00.232483 4926 cpu_manager.go:414] "containerMap: removing container" podUID="e2bce597-c86b-4665-88af-ac36651a05b1" containerName="revoke-keys" Oct 30 22:15:00.232602 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:00.232507 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2bce597-c86b-4665-88af-ac36651a05b1" containerName="revoke-keys" Oct 30 22:15:00.232660 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:00.232631 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2bce597-c86b-4665-88af-ac36651a05b1" containerName="revoke-keys" Oct 30 22:15:00.236069 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:00.236022 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg" Oct 30 22:15:00.255161 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:00.255132 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg"] Oct 30 22:15:00.260414 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:00.260384 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57h8p\" (UniqueName: \"kubernetes.io/projected/24f42e96-7ae1-43b5-8854-6b7887dda105-kube-api-access-57h8p\") pod \"maas-api-key-cleanup-29889975-7zsfg\" (UID: \"24f42e96-7ae1-43b5-8854-6b7887dda105\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg" Oct 30 22:15:00.260589 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:00.260421 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f42e96-7ae1-43b5-8854-6b7887dda105-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-7zsfg\" (UID: \"24f42e96-7ae1-43b5-8854-6b7887dda105\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg" Oct 30 22:15:00.262858 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:00.262834 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mssmb\"" Oct 30 22:15:00.262998 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:00.262880 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 22:15:00.262998 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:00.262909 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 22:15:00.361451 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:00.361410 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57h8p\" (UniqueName: \"kubernetes.io/projected/24f42e96-7ae1-43b5-8854-6b7887dda105-kube-api-access-57h8p\") pod \"maas-api-key-cleanup-29889975-7zsfg\" (UID: \"24f42e96-7ae1-43b5-8854-6b7887dda105\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg" Oct 30 22:15:00.361451 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:00.361451 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f42e96-7ae1-43b5-8854-6b7887dda105-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-7zsfg\" (UID: \"24f42e96-7ae1-43b5-8854-6b7887dda105\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg" Oct 30 22:15:00.362209 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:00.362190 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f42e96-7ae1-43b5-8854-6b7887dda105-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-7zsfg\" (UID: \"24f42e96-7ae1-43b5-8854-6b7887dda105\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg" Oct 30 22:15:00.380464 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:00.380429 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57h8p\" (UniqueName: \"kubernetes.io/projected/24f42e96-7ae1-43b5-8854-6b7887dda105-kube-api-access-57h8p\") pod \"maas-api-key-cleanup-29889975-7zsfg\" (UID: \"24f42e96-7ae1-43b5-8854-6b7887dda105\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg" Oct 30 22:15:00.551157 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:00.551064 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg" Oct 30 22:15:00.847925 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:00.847829 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg"] Oct 30 22:15:00.850973 ip-10-0-142-3 kubenswrapper[4926]: W1030 22:15:00.850934 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f42e96_7ae1_43b5_8854_6b7887dda105.slice/crio-7a356fdd719998c04a35dd8fde6ead1df1b45e57ed5aadb04cde8f01b640cdc1 WatchSource:0}: Error finding container 7a356fdd719998c04a35dd8fde6ead1df1b45e57ed5aadb04cde8f01b640cdc1: Status 404 returned error can't find the container with id 7a356fdd719998c04a35dd8fde6ead1df1b45e57ed5aadb04cde8f01b640cdc1 Oct 30 22:15:01.171429 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:01.171391 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg" event={"ID":"24f42e96-7ae1-43b5-8854-6b7887dda105","Type":"ContainerStarted","Data":"7a356fdd719998c04a35dd8fde6ead1df1b45e57ed5aadb04cde8f01b640cdc1"} Oct 30 22:15:02.179235 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:02.179196 4926 generic.go:356] "Generic (PLEG): container finished" podID="24f42e96-7ae1-43b5-8854-6b7887dda105" containerID="0ba2e8d2aaa41faf23ca33bbeaa3e0110b241404db576325a2a6ce14a8aed175" exitCode=60 Oct 30 22:15:02.179654 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:02.179281 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg" event={"ID":"24f42e96-7ae1-43b5-8854-6b7887dda105","Type":"ContainerDied","Data":"0ba2e8d2aaa41faf23ca33bbeaa3e0110b241404db576325a2a6ce14a8aed175"} Oct 30 22:15:02.179654 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:02.179484 4926 scope.go:122] "RemoveContainer" containerID="0ba2e8d2aaa41faf23ca33bbeaa3e0110b241404db576325a2a6ce14a8aed175" Oct 30 22:15:02.934900 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:02.934864 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-75ttb"] Oct 30 22:15:03.188112 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:03.187995 4926 generic.go:356] "Generic (PLEG): container finished" podID="24f42e96-7ae1-43b5-8854-6b7887dda105" containerID="0f426900fcdcb5d8cc3f59f7eefe424e4c40756957ba984ea04860dd9d3535b3" exitCode=60 Oct 30 22:15:03.188112 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:03.188068 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg" event={"ID":"24f42e96-7ae1-43b5-8854-6b7887dda105","Type":"ContainerDied","Data":"0f426900fcdcb5d8cc3f59f7eefe424e4c40756957ba984ea04860dd9d3535b3"} Oct 30 22:15:03.188633 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:03.188142 4926 scope.go:122] "RemoveContainer" containerID="0ba2e8d2aaa41faf23ca33bbeaa3e0110b241404db576325a2a6ce14a8aed175" Oct 30 22:15:03.188633 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:03.188412 4926 scope.go:122] "RemoveContainer" containerID="0f426900fcdcb5d8cc3f59f7eefe424e4c40756957ba984ea04860dd9d3535b3" Oct 30 22:15:03.188737 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:15:03.188676 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-7zsfg_odh-ai-gateway-infra(24f42e96-7ae1-43b5-8854-6b7887dda105)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg" podUID="24f42e96-7ae1-43b5-8854-6b7887dda105" Oct 30 22:15:04.235056 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:04.234743 4926 scope.go:122] "RemoveContainer" containerID="0f426900fcdcb5d8cc3f59f7eefe424e4c40756957ba984ea04860dd9d3535b3" Oct 30 22:15:04.244077 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:15:04.242066 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-7zsfg_odh-ai-gateway-infra(24f42e96-7ae1-43b5-8854-6b7887dda105)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg" podUID="24f42e96-7ae1-43b5-8854-6b7887dda105" Oct 30 22:15:07.483341 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:15:07.483303 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:15:13.656428 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:13.656395 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-pz9vv_a3947186-e2e4-40cb-8a88-07276844926f/manager/0.log" Oct 30 22:15:13.862764 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:13.862731 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-mq7lz_0b9f291b-8192-4185-82d3-fd195ba1c62d/manager/0.log" Oct 30 22:15:14.141451 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:14.141419 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 22:15:14.149222 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:14.149194 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 22:15:14.427974 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:14.427946 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-dcd6m_0a14816b-84fb-4f2f-aa6d-b5b23beb7a6e/server/0.log" Oct 30 22:15:16.007800 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:16.007768 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29889975-7zsfg_24f42e96-7ae1-43b5-8854-6b7887dda105/cleanup/1.log" Oct 30 22:15:16.008346 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:16.008327 4926 scope.go:122] "RemoveContainer" containerID="0f426900fcdcb5d8cc3f59f7eefe424e4c40756957ba984ea04860dd9d3535b3" Oct 30 22:15:16.199169 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:16.199137 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-njcsj_db0d1f7b-d853-4fad-a3bf-3075b28c8833/revoke-keys/0.log" Oct 30 22:15:16.302402 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:16.302364 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg" event={"ID":"24f42e96-7ae1-43b5-8854-6b7887dda105","Type":"ContainerStarted","Data":"c0dfb942f2c367cf1bcf68e1ab27ae38406cb7a754f282deb11f7e3155e36fcc"} Oct 30 22:15:16.340277 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:16.340220 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg" podStartSLOduration=16.340207114 podStartE2EDuration="16.340207114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:16.339231689 +0000 UTC m=+4503.460305437" watchObservedRunningTime="2026-10-30 22:15:16.340207114 +0000 UTC m=+4503.461280860" Oct 30 22:15:16.767992 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:16.767957 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-p5f66_e2bce597-c86b-4665-88af-ac36651a05b1/revoke-keys/0.log" Oct 30 22:15:17.137275 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:17.137184 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-v7k28_3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60/revoke-keys/0.log" Oct 30 22:15:17.310044 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:17.309996 4926 generic.go:356] "Generic (PLEG): container finished" podID="24f42e96-7ae1-43b5-8854-6b7887dda105" containerID="c0dfb942f2c367cf1bcf68e1ab27ae38406cb7a754f282deb11f7e3155e36fcc" exitCode=60 Oct 30 22:15:17.310044 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:17.310023 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg" event={"ID":"24f42e96-7ae1-43b5-8854-6b7887dda105","Type":"ContainerDied","Data":"c0dfb942f2c367cf1bcf68e1ab27ae38406cb7a754f282deb11f7e3155e36fcc"} Oct 30 22:15:17.310306 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:17.310099 4926 scope.go:122] "RemoveContainer" containerID="0f426900fcdcb5d8cc3f59f7eefe424e4c40756957ba984ea04860dd9d3535b3" Oct 30 22:15:17.310423 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:17.310403 4926 scope.go:122] "RemoveContainer" containerID="c0dfb942f2c367cf1bcf68e1ab27ae38406cb7a754f282deb11f7e3155e36fcc" Oct 30 22:15:17.310694 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:15:17.310669 4926 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-7zsfg_odh-ai-gateway-infra(24f42e96-7ae1-43b5-8854-6b7887dda105)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg" podUID="24f42e96-7ae1-43b5-8854-6b7887dda105" Oct 30 22:15:17.355521 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:17.355487 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg"] Oct 30 22:15:17.745826 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:15:17.745786 4926 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f426900fcdcb5d8cc3f59f7eefe424e4c40756957ba984ea04860dd9d3535b3\": container with ID starting with 0f426900fcdcb5d8cc3f59f7eefe424e4c40756957ba984ea04860dd9d3535b3 not found: ID does not exist" containerID="0f426900fcdcb5d8cc3f59f7eefe424e4c40756957ba984ea04860dd9d3535b3" Oct 30 22:15:18.450562 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:18.450537 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg" Oct 30 22:15:18.573195 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:18.573116 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24f42e96-7ae1-43b5-8854-6b7887dda105-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f42e96-7ae1-43b5-8854-6b7887dda105-maas-api-service-ca\") pod \"24f42e96-7ae1-43b5-8854-6b7887dda105\" (UID: \"24f42e96-7ae1-43b5-8854-6b7887dda105\") " Oct 30 22:15:18.573404 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:18.573284 4926 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f42e96-7ae1-43b5-8854-6b7887dda105-kube-api-access-57h8p\" (UniqueName: \"kubernetes.io/projected/24f42e96-7ae1-43b5-8854-6b7887dda105-kube-api-access-57h8p\") pod \"24f42e96-7ae1-43b5-8854-6b7887dda105\" (UID: \"24f42e96-7ae1-43b5-8854-6b7887dda105\") " Oct 30 22:15:18.573618 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:18.573588 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f42e96-7ae1-43b5-8854-6b7887dda105-maas-api-service-ca" pod "24f42e96-7ae1-43b5-8854-6b7887dda105" (UID: "24f42e96-7ae1-43b5-8854-6b7887dda105"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:18.575404 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:18.575375 4926 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f42e96-7ae1-43b5-8854-6b7887dda105-kube-api-access-57h8p" pod "24f42e96-7ae1-43b5-8854-6b7887dda105" (UID: "24f42e96-7ae1-43b5-8854-6b7887dda105"). InnerVolumeSpecName "kube-api-access-57h8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:18.674950 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:18.674913 4926 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57h8p\" (UniqueName: \"kubernetes.io/projected/24f42e96-7ae1-43b5-8854-6b7887dda105-kube-api-access-57h8p\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:15:18.674950 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:18.674950 4926 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f42e96-7ae1-43b5-8854-6b7887dda105-maas-api-service-ca\") on node \"ip-10-0-142-3.ec2.internal\" DevicePath \"\"" Oct 30 22:15:19.239913 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:19.239878 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-jgd2h_cd34e890-4d69-4266-8aed-c976f7c5259d/manager/0.log" Oct 30 22:15:19.324586 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:19.324496 4926 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg" Oct 30 22:15:19.324586 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:19.324532 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg" event={"ID":"24f42e96-7ae1-43b5-8854-6b7887dda105","Type":"ContainerDied","Data":"7a356fdd719998c04a35dd8fde6ead1df1b45e57ed5aadb04cde8f01b640cdc1"} Oct 30 22:15:19.324854 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:19.324605 4926 scope.go:122] "RemoveContainer" containerID="c0dfb942f2c367cf1bcf68e1ab27ae38406cb7a754f282deb11f7e3155e36fcc" Oct 30 22:15:19.372482 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:19.372453 4926 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg"] Oct 30 22:15:19.383465 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:19.383438 4926 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-7zsfg"] Oct 30 22:15:19.427951 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:19.427913 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-qghvs_35046001-5dfb-4df4-b0bb-602292f94643/manager/0.log" Oct 30 22:15:19.826102 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:19.826066 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-ngzg7_e8fe515a-6a45-4d3c-9715-4a156d26a237/manager/0.log" Oct 30 22:15:20.107578 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:20.107497 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-75ttb_69012155-4a19-4aed-a956-28b5efd43cf1/limitador/0.log" Oct 30 22:15:20.260474 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:20.260443 4926 scope.go:122] "RemoveContainer" containerID="796dac2e2b5584de0e13aa4aa3e410578e0809d509f17dbf4dd004a478883236" Oct 30 22:15:20.299072 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:20.299019 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-sw8wh_1fa6ac28-3aaa-4c2c-9220-a1fea9494c87/manager/0.log" Oct 30 22:15:21.015252 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:21.015220 4926 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24f42e96-7ae1-43b5-8854-6b7887dda105" path="/var/lib/kubelet/pods/24f42e96-7ae1-43b5-8854-6b7887dda105/volumes" Oct 30 22:15:21.031091 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:21.031059 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-669569fcd5lntp7_4f9713f2-a62e-49a0-b9b8-7bfb6b828f35/istio-proxy/0.log" Oct 30 22:15:21.841656 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:21.841628 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-bpsh4_36be1c6a-ce91-4582-afef-56c4d74c0335/istio-proxy/0.log" Oct 30 22:15:23.232621 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:23.232585 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-rh5rw_ed448351-dac3-4d81-9e30-bce9475607eb/istio-proxy/0.log" Oct 30 22:15:23.434502 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:23.434471 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-wxstq_d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94/istio-proxy/1.log" Oct 30 22:15:23.822643 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:23.822607 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b_b3d0842d-2236-4cad-ad5a-f1c47655c5be/istio-proxy/0.log" Oct 30 22:15:24.387010 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:24.386972 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6789dbc4b5-5gqgl_276d4076-cd5a-45d6-b4f8-2de14f9c9a0d/router/0.log" Oct 30 22:15:24.845799 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:24.845764 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-wxstq_d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94/istio-proxy/0.log" Oct 30 22:15:36.904418 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:36.904375 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-pz9vv_a3947186-e2e4-40cb-8a88-07276844926f/manager/0.log" Oct 30 22:15:37.114331 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:37.114292 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-mq7lz_0b9f291b-8192-4185-82d3-fd195ba1c62d/manager/0.log" Oct 30 22:15:37.484208 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:15:37.484167 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:15:37.581722 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:37.581685 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-dcd6m_0a14816b-84fb-4f2f-aa6d-b5b23beb7a6e/server/0.log" Oct 30 22:15:38.903170 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:38.903137 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-njcsj_db0d1f7b-d853-4fad-a3bf-3075b28c8833/revoke-keys/0.log" Oct 30 22:15:39.275647 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:39.275612 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-p5f66_e2bce597-c86b-4665-88af-ac36651a05b1/revoke-keys/0.log" Oct 30 22:15:39.649821 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:39.649735 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-v7k28_3f5611dd-1dcb-4d4d-b5e1-7ba153c4cd60/revoke-keys/0.log" Oct 30 22:15:41.217635 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:41.217600 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-jgd2h_cd34e890-4d69-4266-8aed-c976f7c5259d/manager/0.log" Oct 30 22:15:41.404482 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:41.404438 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-qghvs_35046001-5dfb-4df4-b0bb-602292f94643/manager/0.log" Oct 30 22:15:41.785489 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:41.785449 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-ngzg7_e8fe515a-6a45-4d3c-9715-4a156d26a237/manager/0.log" Oct 30 22:15:42.008759 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:42.008727 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-75ttb_69012155-4a19-4aed-a956-28b5efd43cf1/limitador/0.log" Oct 30 22:15:42.209703 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:42.209666 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-sw8wh_1fa6ac28-3aaa-4c2c-9220-a1fea9494c87/manager/0.log" Oct 30 22:15:42.744656 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:42.744624 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-669569fcd5lntp7_4f9713f2-a62e-49a0-b9b8-7bfb6b828f35/istio-proxy/0.log" Oct 30 22:15:43.568385 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:43.568353 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-bpsh4_36be1c6a-ce91-4582-afef-56c4d74c0335/istio-proxy/0.log" Oct 30 22:15:44.969643 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:44.969604 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-rh5rw_ed448351-dac3-4d81-9e30-bce9475607eb/istio-proxy/0.log" Oct 30 22:15:45.175512 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:45.175476 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-wxstq_d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94/istio-proxy/1.log" Oct 30 22:15:45.567275 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:45.567243 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b_b3d0842d-2236-4cad-ad5a-f1c47655c5be/istio-proxy/0.log" Oct 30 22:15:46.139010 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:15:46.138972 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6789dbc4b5-5gqgl_276d4076-cd5a-45d6-b4f8-2de14f9c9a0d/router/0.log" Oct 30 22:16:07.482662 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:16:07.482624 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:16:20.288398 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:16:20.288357 4926 scope.go:122] "RemoveContainer" containerID="92b54bc35fcf6b44a0e07ba94f9cc49cfbc6a2fc6dfc11af0f20037a2c51e06c" Oct 30 22:16:37.484731 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:16:37.484690 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:17:05.520077 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:05.520019 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-47t7f/must-gather-shvgw"] Oct 30 22:17:05.520607 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:05.520455 4926 cpu_manager.go:414] "containerMap: removing container" podUID="24f42e96-7ae1-43b5-8854-6b7887dda105" containerName="cleanup" Oct 30 22:17:05.520607 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:05.520467 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f42e96-7ae1-43b5-8854-6b7887dda105" containerName="cleanup" Oct 30 22:17:05.520607 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:05.520479 4926 cpu_manager.go:414] "containerMap: removing container" podUID="24f42e96-7ae1-43b5-8854-6b7887dda105" containerName="cleanup" Oct 30 22:17:05.520607 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:05.520484 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f42e96-7ae1-43b5-8854-6b7887dda105" containerName="cleanup" Oct 30 22:17:05.520607 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:05.520557 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f42e96-7ae1-43b5-8854-6b7887dda105" containerName="cleanup" Oct 30 22:17:05.520607 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:05.520564 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f42e96-7ae1-43b5-8854-6b7887dda105" containerName="cleanup" Oct 30 22:17:05.520826 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:05.520632 4926 cpu_manager.go:414] "containerMap: removing container" podUID="24f42e96-7ae1-43b5-8854-6b7887dda105" containerName="cleanup" Oct 30 22:17:05.520826 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:05.520637 4926 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f42e96-7ae1-43b5-8854-6b7887dda105" containerName="cleanup" Oct 30 22:17:05.520826 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:05.520715 4926 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f42e96-7ae1-43b5-8854-6b7887dda105" containerName="cleanup" Oct 30 22:17:05.523721 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:05.523702 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47t7f/must-gather-shvgw" Oct 30 22:17:05.549609 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:05.549577 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-47t7f\"/\"openshift-service-ca.crt\"" Oct 30 22:17:05.549609 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:05.549600 4926 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-47t7f\"/\"default-dockercfg-5hv7n\"" Oct 30 22:17:05.549847 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:05.549671 4926 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-47t7f\"/\"kube-root-ca.crt\"" Oct 30 22:17:05.552747 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:05.552718 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47t7f/must-gather-shvgw"] Oct 30 22:17:05.656265 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:05.656213 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tpfs\" (UniqueName: \"kubernetes.io/projected/cb3926e0-ac51-47b0-bf9c-8d9a3cfa12b1-kube-api-access-2tpfs\") pod \"must-gather-shvgw\" (UID: \"cb3926e0-ac51-47b0-bf9c-8d9a3cfa12b1\") " pod="openshift-must-gather-47t7f/must-gather-shvgw" Oct 30 22:17:05.656467 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:05.656339 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cb3926e0-ac51-47b0-bf9c-8d9a3cfa12b1-must-gather-output\") pod \"must-gather-shvgw\" (UID: \"cb3926e0-ac51-47b0-bf9c-8d9a3cfa12b1\") " pod="openshift-must-gather-47t7f/must-gather-shvgw" Oct 30 22:17:05.757285 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:05.757241 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tpfs\" (UniqueName: \"kubernetes.io/projected/cb3926e0-ac51-47b0-bf9c-8d9a3cfa12b1-kube-api-access-2tpfs\") pod \"must-gather-shvgw\" (UID: \"cb3926e0-ac51-47b0-bf9c-8d9a3cfa12b1\") " pod="openshift-must-gather-47t7f/must-gather-shvgw" Oct 30 22:17:05.757480 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:05.757325 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cb3926e0-ac51-47b0-bf9c-8d9a3cfa12b1-must-gather-output\") pod \"must-gather-shvgw\" (UID: \"cb3926e0-ac51-47b0-bf9c-8d9a3cfa12b1\") " pod="openshift-must-gather-47t7f/must-gather-shvgw" Oct 30 22:17:05.757723 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:05.757701 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cb3926e0-ac51-47b0-bf9c-8d9a3cfa12b1-must-gather-output\") pod \"must-gather-shvgw\" (UID: \"cb3926e0-ac51-47b0-bf9c-8d9a3cfa12b1\") " pod="openshift-must-gather-47t7f/must-gather-shvgw" Oct 30 22:17:05.780136 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:05.777174 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tpfs\" (UniqueName: \"kubernetes.io/projected/cb3926e0-ac51-47b0-bf9c-8d9a3cfa12b1-kube-api-access-2tpfs\") pod \"must-gather-shvgw\" (UID: \"cb3926e0-ac51-47b0-bf9c-8d9a3cfa12b1\") " pod="openshift-must-gather-47t7f/must-gather-shvgw" Oct 30 22:17:05.838279 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:05.838226 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47t7f/must-gather-shvgw" Oct 30 22:17:06.147671 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:06.147587 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47t7f/must-gather-shvgw"] Oct 30 22:17:06.155464 ip-10-0-142-3 kubenswrapper[4926]: W1030 22:17:06.155433 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb3926e0_ac51_47b0_bf9c_8d9a3cfa12b1.slice/crio-85e4481fc468f278aa2c2f0d49538c9a9024b7e31594e342e58fb0b5c0cbe048 WatchSource:0}: Error finding container 85e4481fc468f278aa2c2f0d49538c9a9024b7e31594e342e58fb0b5c0cbe048: Status 404 returned error can't find the container with id 85e4481fc468f278aa2c2f0d49538c9a9024b7e31594e342e58fb0b5c0cbe048 Oct 30 22:17:06.157726 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:06.157702 4926 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:17:06.939631 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:06.939582 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47t7f/must-gather-shvgw" event={"ID":"cb3926e0-ac51-47b0-bf9c-8d9a3cfa12b1","Type":"ContainerStarted","Data":"85e4481fc468f278aa2c2f0d49538c9a9024b7e31594e342e58fb0b5c0cbe048"} Oct 30 22:17:07.489793 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:17:07.489757 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:17:07.952335 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:07.952299 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47t7f/must-gather-shvgw" event={"ID":"cb3926e0-ac51-47b0-bf9c-8d9a3cfa12b1","Type":"ContainerStarted","Data":"7dec06b3778704ec8b314b6005296b3b01766421bedc53598f3e8d33690e804e"} Oct 30 22:17:07.952335 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:07.952343 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47t7f/must-gather-shvgw" event={"ID":"cb3926e0-ac51-47b0-bf9c-8d9a3cfa12b1","Type":"ContainerStarted","Data":"d3819c4a052ad5f0877da8c9241246646d731e11e41f77bd5aff54a14acdf40d"} Oct 30 22:17:07.989345 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:07.989281 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-47t7f/must-gather-shvgw" podStartSLOduration=2.257757265 podStartE2EDuration="2.989267228s" totalImagesPullingTime="731.509963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:07.987853897 +0000 UTC m=+4615.108927643" watchObservedRunningTime="2026-10-30 22:17:07.989267228 +0000 UTC m=+4615.110340975" Oct 30 22:17:10.258844 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:10.258808 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m9dm5_5ae2e7bf-9e4d-4839-acfc-e2e3ca66df0e/global-pull-secret-syncer/0.log" Oct 30 22:17:10.608019 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:10.607947 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-b9fhn_c0b2deee-2b0c-4c2c-9ac5-debd33f12ba3/connection-checker/0.log" Oct 30 22:17:10.742636 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:10.742587 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-zdvb2_0f0b2878-417e-4986-8067-b50a16d16967/connection-checker/0.log" Oct 30 22:17:10.880983 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:10.880898 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-zr75n_6d9c7935-1d3d-4323-ac72-008a394e7f31/connection-checker/0.log" Oct 30 22:17:11.398099 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:11.398060 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mgsr5_5525d794-7ab8-4d4a-84fa-bfa14d9d3d8e/konnectivity-agent/0.log" Oct 30 22:17:11.794044 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:11.793995 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-3.ec2.internal_10ed65d1de0a53fd4cc159ce5efb836d/haproxy/0.log" Oct 30 22:17:18.406055 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:18.404839 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-jgd2h_cd34e890-4d69-4266-8aed-c976f7c5259d/manager/0.log" Oct 30 22:17:18.549756 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:18.547812 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-qghvs_35046001-5dfb-4df4-b0bb-602292f94643/manager/0.log" Oct 30 22:17:18.856225 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:18.856180 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-ngzg7_e8fe515a-6a45-4d3c-9715-4a156d26a237/manager/0.log" Oct 30 22:17:18.953799 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:18.953580 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-75ttb_69012155-4a19-4aed-a956-28b5efd43cf1/limitador/0.log" Oct 30 22:17:19.217704 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:19.217653 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-sw8wh_1fa6ac28-3aaa-4c2c-9220-a1fea9494c87/manager/0.log" Oct 30 22:17:20.277698 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:20.277657 4926 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg"] Oct 30 22:17:20.286053 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:20.283994 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg" Oct 30 22:17:20.301588 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:20.301553 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg"] Oct 30 22:17:20.411729 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:20.411686 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eadfe4d2-1082-4b31-b806-d7aeac6c6907-sys\") pod \"perf-node-gather-daemonset-bcgzg\" (UID: \"eadfe4d2-1082-4b31-b806-d7aeac6c6907\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg" Oct 30 22:17:20.411923 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:20.411763 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eadfe4d2-1082-4b31-b806-d7aeac6c6907-proc\") pod \"perf-node-gather-daemonset-bcgzg\" (UID: \"eadfe4d2-1082-4b31-b806-d7aeac6c6907\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg" Oct 30 22:17:20.411923 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:20.411826 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zfgl\" (UniqueName: \"kubernetes.io/projected/eadfe4d2-1082-4b31-b806-d7aeac6c6907-kube-api-access-4zfgl\") pod \"perf-node-gather-daemonset-bcgzg\" (UID: \"eadfe4d2-1082-4b31-b806-d7aeac6c6907\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg" Oct 30 22:17:20.411923 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:20.411847 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eadfe4d2-1082-4b31-b806-d7aeac6c6907-podres\") pod \"perf-node-gather-daemonset-bcgzg\" (UID: \"eadfe4d2-1082-4b31-b806-d7aeac6c6907\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg" Oct 30 22:17:20.412137 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:20.411924 4926 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eadfe4d2-1082-4b31-b806-d7aeac6c6907-lib-modules\") pod \"perf-node-gather-daemonset-bcgzg\" (UID: \"eadfe4d2-1082-4b31-b806-d7aeac6c6907\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg" Oct 30 22:17:20.513286 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:20.513237 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zfgl\" (UniqueName: \"kubernetes.io/projected/eadfe4d2-1082-4b31-b806-d7aeac6c6907-kube-api-access-4zfgl\") pod \"perf-node-gather-daemonset-bcgzg\" (UID: \"eadfe4d2-1082-4b31-b806-d7aeac6c6907\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg" Oct 30 22:17:20.513498 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:20.513300 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eadfe4d2-1082-4b31-b806-d7aeac6c6907-podres\") pod \"perf-node-gather-daemonset-bcgzg\" (UID: \"eadfe4d2-1082-4b31-b806-d7aeac6c6907\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg" Oct 30 22:17:20.513498 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:20.513341 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eadfe4d2-1082-4b31-b806-d7aeac6c6907-lib-modules\") pod \"perf-node-gather-daemonset-bcgzg\" (UID: \"eadfe4d2-1082-4b31-b806-d7aeac6c6907\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg" Oct 30 22:17:20.513498 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:20.513449 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eadfe4d2-1082-4b31-b806-d7aeac6c6907-podres\") pod \"perf-node-gather-daemonset-bcgzg\" (UID: \"eadfe4d2-1082-4b31-b806-d7aeac6c6907\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg" Oct 30 22:17:20.513498 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:20.513480 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eadfe4d2-1082-4b31-b806-d7aeac6c6907-sys\") pod \"perf-node-gather-daemonset-bcgzg\" (UID: \"eadfe4d2-1082-4b31-b806-d7aeac6c6907\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg" Oct 30 22:17:20.513711 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:20.513506 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eadfe4d2-1082-4b31-b806-d7aeac6c6907-lib-modules\") pod \"perf-node-gather-daemonset-bcgzg\" (UID: \"eadfe4d2-1082-4b31-b806-d7aeac6c6907\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg" Oct 30 22:17:20.513711 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:20.513512 4926 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eadfe4d2-1082-4b31-b806-d7aeac6c6907-proc\") pod \"perf-node-gather-daemonset-bcgzg\" (UID: \"eadfe4d2-1082-4b31-b806-d7aeac6c6907\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg" Oct 30 22:17:20.513711 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:20.513567 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eadfe4d2-1082-4b31-b806-d7aeac6c6907-proc\") pod \"perf-node-gather-daemonset-bcgzg\" (UID: \"eadfe4d2-1082-4b31-b806-d7aeac6c6907\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg" Oct 30 22:17:20.513711 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:20.513516 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eadfe4d2-1082-4b31-b806-d7aeac6c6907-sys\") pod \"perf-node-gather-daemonset-bcgzg\" (UID: \"eadfe4d2-1082-4b31-b806-d7aeac6c6907\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg" Oct 30 22:17:20.534795 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:20.534727 4926 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zfgl\" (UniqueName: \"kubernetes.io/projected/eadfe4d2-1082-4b31-b806-d7aeac6c6907-kube-api-access-4zfgl\") pod \"perf-node-gather-daemonset-bcgzg\" (UID: \"eadfe4d2-1082-4b31-b806-d7aeac6c6907\") " pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg" Oct 30 22:17:20.606576 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:20.606530 4926 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg" Oct 30 22:17:20.923473 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:20.922444 4926 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg"] Oct 30 22:17:20.925915 ip-10-0-142-3 kubenswrapper[4926]: W1030 22:17:20.925884 4926 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podeadfe4d2_1082_4b31_b806_d7aeac6c6907.slice/crio-6c021a8243fa1126738625f2d6f2d4894d7906db5419fc54e729b90abbb5a802 WatchSource:0}: Error finding container 6c021a8243fa1126738625f2d6f2d4894d7906db5419fc54e729b90abbb5a802: Status 404 returned error can't find the container with id 6c021a8243fa1126738625f2d6f2d4894d7906db5419fc54e729b90abbb5a802 Oct 30 22:17:21.149057 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:21.145011 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg" event={"ID":"eadfe4d2-1082-4b31-b806-d7aeac6c6907","Type":"ContainerStarted","Data":"ae7f3be3a552e3b6b61918370d810a840f9d31de1e62e1f9e10b42d405a59b97"} Oct 30 22:17:21.149057 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:21.145087 4926 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg" event={"ID":"eadfe4d2-1082-4b31-b806-d7aeac6c6907","Type":"ContainerStarted","Data":"6c021a8243fa1126738625f2d6f2d4894d7906db5419fc54e729b90abbb5a802"} Oct 30 22:17:21.149057 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:21.145797 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg" Oct 30 22:17:21.189739 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:21.189635 4926 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg" podStartSLOduration=1.18962054 podStartE2EDuration="1.18962054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:21.188541231 +0000 UTC m=+4628.309614978" watchObservedRunningTime="2026-10-30 22:17:21.18962054 +0000 UTC m=+4628.310694319" Oct 30 22:17:21.489303 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:21.489209 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-v5rrz_0b77d4c7-37eb-4bad-8367-31346289726e/cluster-monitoring-operator/0.log" Oct 30 22:17:23.002796 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:23.002758 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s24tx_34ad4f2a-37ae-4b6c-968c-34a4369fe36c/node-exporter/0.log" Oct 30 22:17:23.123993 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:23.123724 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s24tx_34ad4f2a-37ae-4b6c-968c-34a4369fe36c/kube-rbac-proxy/0.log" Oct 30 22:17:23.243785 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:23.243757 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s24tx_34ad4f2a-37ae-4b6c-968c-34a4369fe36c/init-textfile/0.log" Oct 30 22:17:27.563869 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:27.563651 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-wbrhm_9ffe8e19-6888-4eca-ae9d-9d0fc05ba752/networking-console-plugin/0.log" Oct 30 22:17:28.169743 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:28.169707 4926 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-47t7f/perf-node-gather-daemonset-bcgzg" Oct 30 22:17:28.662849 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:28.662762 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/1.log" Oct 30 22:17:28.670634 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:28.670599 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-c5khs_1a2fb9ac-a905-4e20-b1be-7928d5e7881a/console-operator/2.log" Oct 30 22:17:31.130952 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:31.130923 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-87c42_a66e14f7-3c72-48a5-88e8-e9f78a7c7ff5/volume-data-source-validator/0.log" Oct 30 22:17:32.962846 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:32.962811 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-669569fcd5lntp7_4f9713f2-a62e-49a0-b9b8-7bfb6b828f35/istio-proxy/0.log" Oct 30 22:17:33.516775 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:33.516746 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-888xx_662ead00-b524-4a6e-815b-c3399b3c9e6c/dns/0.log" Oct 30 22:17:33.635425 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:33.635394 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-888xx_662ead00-b524-4a6e-815b-c3399b3c9e6c/kube-rbac-proxy/0.log" Oct 30 22:17:33.887943 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:33.887862 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-bpsh4_36be1c6a-ce91-4582-afef-56c4d74c0335/istio-proxy/0.log" Oct 30 22:17:34.268608 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:34.268568 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-54qxj_a1d0e9c5-2d37-4adf-a4a8-b4da4fbaae31/dns-node-resolver/0.log" Oct 30 22:17:34.360330 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:34.360300 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-rh5rw_ed448351-dac3-4d81-9e30-bce9475607eb/istio-proxy/0.log" Oct 30 22:17:34.426055 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:34.425989 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-wxstq_d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94/istio-proxy/0.log" Oct 30 22:17:34.429611 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:34.429587 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-57f78bcd55-wxstq_d9bc6398-ec05-4cb3-8ba6-3dcfc5b9cb94/istio-proxy/1.log" Oct 30 22:17:34.530147 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:34.530066 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-749f59598c-llt9b_b3d0842d-2236-4cad-ad5a-f1c47655c5be/istio-proxy/0.log" Oct 30 22:17:34.741303 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:34.741267 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6789dbc4b5-5gqgl_276d4076-cd5a-45d6-b4f8-2de14f9c9a0d/router/0.log" Oct 30 22:17:35.780755 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:35.780720 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cnj4q_1362c712-7332-4a6e-a215-633d2f02acca/node-ca/0.log" Oct 30 22:17:37.484868 ip-10-0-142-3 kubenswrapper[4926]: E1030 22:17:37.484819 4926 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-q9bnp" containerName="extractor" Oct 30 22:17:38.067761 ip-10-0-142-3 kubenswrapper[4926]: I1030 22:17:38.067726 4926 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-669569fcd5lntp7_4f9713f2-a62e-49a0-b9b8-7bfb6b828f35/istio-proxy/0.log"